Before We Get Started
For this example we want to cover just one specific real-world workflow. When continuously developing, changes need to be pushed out regularly to a staging server where the quality assurance team can review the features. This is just one part of the overall process but an important one to get working well. So in the end, we want Buddy to solve two things:
- Run our unit test suite against the application on every push to
master
branch. - Deploy our code to a server whenever code is pushed to
staging
branch.
To begin with, we first need to make some assumptions about our starting point:
You Have a Testable Application
We are going to assume you have a functioning Laravel application, with unit tests written using PHPUnit. If you have done nothing more than installed a fresh copy of the Laravel application then this prerequisite is met. Even if you do not have the actual unit tests written, just having the unit test runner ready to run tests is sufficient. If you use another testing framework, our examples can be easily adapted for your favorite.
You Have a Working Server
Additionally we will assume you have proper hosting infrastructure setup and configured on a public server running as a production environment. Any hosting will do but our recommendation is always Digital Ocean or Amazon Web Services. If you need to get up and running quickly sign up for Laravel Forge to provision a solid production server in your Digital Ocean account.
You Have a Buddy Account
Next you should have signed up with Buddy. Buddy offers a variety of Buddy Cloud (SaaS) plans starting with a Free tier, while also providing a self-hosted, on-premise version called Buddy GO. For the purposes of this demo, any of the free offerings will suffice, but if you need to add additional users, or need multiple projects to build concurrently, you should look into some of the paid plans.
You Have a Hosted Repository
Finally, your code must be in a Git repository. Something that makes Buddy unique is that it can host repositories itself, or integrate with Github, Gitlab, and Bitbucket – the three most popular repository hosts available. Either solution works fine, although for our demo we have elected to let Buddy host the repository.
Create the Project
To start things off, you will first need to create a project, select your repository hosting service, and do your first push. We chose Buddy to host our code base and named our app "My Laravel Application". You can name it what you like.
Pipelines and Actions
The foundation of Buddy's utility – and continuous development in general – is a workflow driven by pipelines. Pipelines are a series of steps (or "actions") which run against your repository in sequence. What makes Buddy dramatically faster than many of its competitors is that:
- Actions are run in Docker containers, so launching the environment for an action can be done very quickly, especially on subsequent runs.
- Actions share a filesystem common to all actions within a pipeline and is persisted even across subsequent runs of the pipeline. This means, for example, dependencies usually only need to be downloaded once, unless they are updated. You can manually create and edit files in a pipeline. This makes the filesystem useful for storing environment configurations that can be safely stored within Buddy, without having to commit them to source control.
Create a PHP Container
For our first pipeline, let's run our unit tests. For this, we will need a Docker container with PHP installed. This container will use the filesystem that has our repository's code on it as the basis for the test runner – PHPUnit. Buddy of course provides the PHP container option as a build type action.
Buddy's default PHP container can support just about any version of PHP 5.6 or 7.
If you needed something like HHVM then use the Ubuntu container and install the
dependencies yourself or choose a publically hosted HHVM docker container image.
We chose PHP 7.0. Buddy automatically installs Composer
and provides composer install
as a default command to be executed. While this
works for basic setups, a production pipeline needs a bit of fine-tuning. We are
going to make an adjustment to the Run Command settings.
Customize the Run Command
We added vendor/bin/phpunit
to the command being run. We use the version of phpunit
installed in vendor
so that the repository can control which version of PHPUnit
to install and we do not have to worry about global dependencies. Some developers
choose to configure composer test
in their composer.json
file to indicate the
testing commands that should be ran. This is a good alternative and one that is
adopted by other languages such as Node.js which uses npm test
.
Tweak the Container for Performance
Switching to the Packages & Caches
tab, there are a few extra settings we should change.
These settings are provided below for copy-pasting:
apt-get update && apt-get install git zip unzip -y
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
We have done two things in this panel to help improve the performance of Composer.
- We added the
zip
andunzip
packages to the container. These will allow Composer to download packages much more quickly than the fallback of using Git. - We have added a
.composer
directory to the action's cache. This is the directory where Composer will cache the zip files, so that on subsequent runs it will not have to do download them again. The path here can be tricky and must be exact. The.composer
directory is created within the pipeline's mounted fileystem path and must be absolute. If you change the mountpoint path or your project name is different than ours you'll need to change the/my-laravel-app/.composer/
path appropriately. Also remember that we want to cache the entire.composer
directory so the trailing slash is required per the filesystem's pattern matching rules.
Give the Action a Name
Now's as good a time as any to provide a familiar name for the Action. We chose "Run PHPUnit" as that's what it does. If your testing commands are different then perhaps a more generic "Run Composer Test" is more accurate.
When creating your first action for a new project, the action is added to a pipeline that is also created for you. When adding new actions to an existing pipeline, they are automatically added for you. So after clicking "Add this action" you should be presented with options to add more actions or proceed to viewing your pipeline.
Verify the Pipeline Settings
Before we try things out, we are going to verify a few more things.
- Go to the
Pipeline Settings
page to change your pipeline's name. By default it's something silly like "My First Pipeline". In our case we named it after the primary action, "Run Unit Tests". - Also verify the Trigger Mode is set to "On every push" and for the
master
branch. For our testing pipeline this is perfect but for other pipelines you'll want to configure these settings appropriately.
Setup the Environment
We are going to use Buddy's ability to manipulate the pipeline filesystem to
create a .env
file for our unit testing environment. From the Pipeline Settings
page, click in the right sidebar "Browse the filesystem" to do just that. If you
have a .env
file on your local machine you can upload it directly, or you can
use Buddy's interface to create one directly based on the .env.example
default.
We used the .env.example
file that ships with Laravel, but with an APP_KEY
already generated. Depending on the complexity of your unit tests, you may need
to tweak database settings or modify other parameters. Also remember to verify
your test runner's configs (in the case of PHPUnit that would be phpunit.xml
).
Run the Pipeline Manually
At this point, you can run the pipeline manually. Assuming everything went to plan, your dependencies should download, PHPunit should run your tests, and you should get a happy green build icon.
The first time you run the pipeline, the container is setup. This can take a bit
of time so be patient. If the pipeline executed without a problem, try executing
it again and you'll find it to be much faster. Remember that Composer outputs a
composer.lock
file in your directory which will skip installation of dependencies
on the second build. For this reason, you would want to either delete the lock file
manually through the filesystem browser should you want to install again, or better
yet, commit the composer.lock
file with your repository by removing the line
from the .gitignore
file in the root of the Laravel code base. This will ensure
the right version of all dependencies are installed and consistently across builds.
Build and Deploy the Application
Now that we have a functioning unit test pipeline, let's use Buddy to automate our deployment process. The basic building blocks are going to remain the same, but there are going to be more actions involved, and we are going to use connect two pipelines together to accomplish the over all workflow.
Create a Staging Pipeline
Our first deployment pipeline is going to use a manual trigger to build the
application, including all its dependencies, and upload them to our previously
setup staging server. We named the pipeline "Deploy to Staging" and setup the
Trigger Mode to be Manual
on a Specific branch
which in our case is stage
.
Build PHP Application Action
Our pipeline is going to be made up of five actions. First, we need a PHP container-based action to install all the application's dependencies. We named it "Build PHP application". The Run Commands we used are as follows:
composer install --no-dev
npm install
npm run prod
Notice that we are skipping the dev dependencies for this pipeline using the
--no-dev
switch. Since we are uploading to a staging server, they should not
be necessary as the dev dependencies are only need to build the app, not run it.
You will also see that we are installing and compiling all our frontend assets using NPM and Gulp. These services are not available in the default PHP container and will require some additions to the container's build steps and caches:
curl -sL https://deb.nodesource.com/setup_6.x | bash -
apt-get update && apt-get install git build-essential nodejs zip unzip -y
npm install --global gulp marked node-gyp
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
In addition to installing Composer, this will install the latest version of Node.js and Laravel's frontend build tools. Much like we previously did with Composer, we are going to add NPM's cache directory to Buddy's list of cached directories to speed up build time:
/my-laravel-app/.composer/
/my-laravel-app/.npm/
Switch to Maintenance Mode Action
If the previous action completes, we have a successful build of the application. Next, we should connect to the staging server and put it into maintenance mode. For this, we will use Buddy's SSH action. The interface for this action includes some handy instructions for installing Buddy's public key on your server. This is necessary so that the pipeline can connect without having to use a password. If you're using Laravel Forge then you can easily upload the key provided directly through the Forge web interface and let Forge do the heavy lifting.
We named the action "Bring Application Down". The Run Command itself is a one-liner:
touch artisan && php artisan down
When we first run the pipeline, artisan
will not have been uploaded to the server,
running touch
will give PHP an (empty) file to try executing. Under the
More Options
section, make sure your working directory on the server is set
correctly, and the action is set to ignore errors. Normally you would want a
pipeline action to stop if any errors are encountered, but for this we can
assume that an error means the application is already down (albeit, less gracefully)
or has not yet been deployed.
Upload Files Action
Now that the application is disabled, we can copy the files to the server. If you are using a hosting provider such as DigitalOcean there are several convenient actions to lookup your available servers which all use the SFTP action under the hood. We are going to be using SFTP action to copy our application files en masse to the staging server. Select the SFTP action, setup your staging server's IP address, port, and credentials just like before, and make sure the remote path is correct. Do not forget to give the actino a good name like "Upload to Staging" which represents exactly what the action does.
There are also several directories within our pipeline we can and you should ignore as uploading them to the staging server is completely unnecessary and will slow down your builds:
/.npm/
/.composer/
/node_modules/
The cache directories for NPM and Composer can be safely ignored. In addition
though, we also exclude the node_modules
directory. Since our assets will have
all been compiled in the first step of this pipeline, we no longer need to have
the contents of node_modules
, and skipping it can dramatically improve upload
times. You could also add /resources/assets/
as npm run prod
should have used
gulp
behind the scenes to build your assets and put them in the public
directory.
The same could be said for directories like /tests/
which are not needed on a
staging server.
It should be noted that this is generally the most time consuming action in the
pipeline and while subsequent runs are generally faster than the first run, the
entire time you are uploading files, your application is down. This makes this
pipeline create some graceful downtime for you application. In practice, deployments
encounter no more than one minute of total maintenance mode down time. That's
pretty neglible for most apps but if it counts, more advanced zero-downtime
strategies can be used with Buddy and faster deployment mechanisms like using
rsync
are possible.
Run Migrations Action
After the files have been uploaded, we can migrate the database. Just like with the "Switch to Maintenance Mode" action, this is an SSH action. You might name the action "Run Migrations" and use the following Run Commands:
php artisan migrate --force
We add the --force
flag so that migrations will be run even though the
application is in a non-local (a staging or production) environment.
Restart Application Action
Now that the application code is deployed, the database is migrated, it's time to bring the application back up out of maintenance mode. You can run Laravel's optimizations and restart queue workers if you have them running. While this could have been done at the same time as the previous step, the best practice approach says that when you have two separate concepts you have two separate processes and therefore separate pipeline actions. The added benefit of following best practices is that Buddy lets you resume any failed action from that point which might lead to resolving failed actions sooner.
We named our action simply "Bring Application Up" with the following Run Commands:
php artisan cache:clear
php artisan config:cache
php artisan route:cache
php artisan optimize
php artisan queue:restart
php artisan up
This should ensure optimal performance from your Laravel application in a production environment.
Setting Up the Environment
Just like with our unit testing pipeline, we need to upload or create a .env
file for our staging environment. It is important to note, however, that this is
the .env
file the staging server will be using, not necessarily what the
actions in the pipeline will use. Make sure your environment variables are set
accordingly and add the .env
file to the pipeline's filesystem.
The resulting pipeline should look something like this:
Pipeline Inception
Buddy can have one pipeline run another pipeline. The pipeline we just created
will not run automatically. That is because we want it to be triggered by a
second pipeline that will be automatically triggered. Even though we run our
unit tests on every push to master
it is a good idea to always run your tests
on staging or production code before deploying it. Thankfully, our unit tests
pipeline is already configured to do this for us, we just need to adjust its
settings and add an additional step. Create another pipeline but from the
Add a new pipeline
form, click Clone existing pipeline
and select your unit
testing pipeline.
Tweak the new pipeline's settings to point to your staging
branch. In this new
pipeline, after the unit testing action has been run, we want to run all the actions
in our previous deployment pipeline. Buddy makes this easy for us: we can run the
deployment pipeline as an action from this testing pipeline – inception style.
This way, once the unit tests have all passed on the staging
branch, your code
will be copied to the server. Create your .env
file as we did with the previous
two pipelines and you are ready to deploy some code – simply by pushing to the
staging
branch. Rinse and repeat these steps to add as many deployment
environments as you like including one for a production
branch.
The initial upload will probably be quite lengthy, however on subsequent commits Buddy is pretty smart about determining what files have changed and need to be copied to the server.
Going Further
With these tools in hand, web artisans can construct powerful and flexible integration and deployment pipelines. We have just begun to scratch the surface of all that you can do with Buddy. There is a wealth of additional features and actions that can be used, such as sending messages to Slack on successful completion or errors, deploying AWS Lambda functions, or building completely custom actions using a generic Ubuntu container. As the tools continue to mature, we look forward to putting them into our own production processes. Ultimately, Buddy makes it easy for the app developers to continuously develop.