AWS Cloud Resume Challenge Pt. 2:
Rebuilding with Terraform
Check out the corresponding github repo here
As I mentioned in my previous porfolio entry, after completing the basic architecture of the website and verifying that everything was working as intended I decided to tear everything to the ground and start fresh, implementing everything top to bottom using Terraform. It is worth noting, in my opinion, that while this was undoubtedly the most challenging aspect of the Cloud Resume Challenge, it was also by far my favorite. It was constantly engaging and I thoroughly enjoyed the freedom it gave to organize the project in a way that made sense to me.
I began by doing some cursory research about Terraform: How does it work? What is the intention behind it's utilization? What are some best-practices? As someone with a background in Audio Engineering, I place a very high value on making my workflows efficient. Templatizing configs and automating legwork was, for me, a hugely important part of my job as a Studio Tech, Sound Designer, and Audio Engineer. When a client is paying you hourly (and especially when they are not) the last thing you want to do is be wasting a bunch of time tinkering with settings and trying to get everything set up in the software just to actually begin the process of recording. Beyond that, the amount of configuration options that go into post-production can be truly staggering. It is extremely easy to go down a rabbit hole and spend 6, 12, 18 hours tweaking software settings, routing and rerouting the signal chain, getting the output and usage levels in the software just right, Et al., which can lead to either unhappy clients for being billed over the estimated hours or wasting my own time on a project-based paying job. As a counter-measure to my own perfectionism and propensity to endlessly explore and tweak settings, I had created a series of templates for different types of projects: full band, solo artist, film sound design, settings for each individual instrument or capture based on the client's vision, etc., as well as facilitating how all/any of those worked in concert to serve the greater project as a whole. For this reason the ideology behind Terraform resonated with me strongly. Work smarter, not harder; do the majority of the possible work up front to streamline the process and prevent log jams downstream (keeping in line with Agile and Lean frameworks).
While researching places to start on this phase of the project, I came across this video by Rishab in Cloud. I found this particularly helpful as a starting point. In it, he covers the Lambda section of the architecture. This includes initializing terraform in your directory, configuring the provider.tf file, packaging the Python code for the function in a Zip file, writing the JSON function that the origin HTML document uses to call the Lambda function URL, configuring and assuming an IAM role for Lambda, and attaching a policy to said role. This was an excellent place to start, to me, as it demonstrated the core concepts and functionality of the software and gave me a solid foundation upon which I could build my infrastructure.
The natural next step was to implement DynamoDB as it is the target of the Lambda function's functionality. This was fairly straightforward to me and did not take much time once I learned the proper resource command.
It was during the creation of my S3 bucket that the monolithic document began to feel a bit unwieldy and disorganized. Surely there are better ways of organizing this project. I spent a fair amount of time at this point reading documentations, blog posts, forum questions, examples, and GitHub repos until I eventually was able to visualize how I wanted to organize the project: Use the main.tf file a a sort of directory hub that would call the different modules I would logically separate by service or function. This was, if not necessarily more efficient from a time perspective for me at this point, a far superior system of organization to the previous iteration which was quickly becoming a rambling and disorganized document. I had to familiarize myself with a lot of new concepts in order to achieve this, not the least of which being that of passing variables in terraform for both security and efficiency purposes, which I achieved primarily by contextualizing other Terraform projects and digging into what the underlying logic was. I took my time on this, as I wanted to be certain that I fully grasped the concepts so as to minimize the amount of re-work I would have to do down the road.
Now, with my S3 bucket, Lambda function, and DynamoDB table in place it was time to work on what I knew would be the most challenging part of the process: Route 53, ACM Certificate creation, DNS Validation, and the creation of the CloudFront Distribution. This section had a LOT of moving parts and was the most challenging part of the entire process by a country mile. Route 53 and the ACM certificate and validation had to be created before the CloudFront distribution could be created, and the CloudFront Distribution had to be created before I could create Alias records pointing my domain name(s) to the CloudFront distribution. This logic pretzel was challenging, but I eventually achieved theoretical* functionality (*more on that later) by removing the Alias resource command from the Route 53 module and creating an "Alias" module that would run seperately, utilizing the depends_on argument, and performing a target apply on the Route 53 module so the certificate information would exist for the creation of the Route 53 records, ACM DNS validation, and creation of the CloudFront Distribution.
I was ready. I double checked my code, my dependencies, made sure all my variables and outputs were being passed properly, and got the beautiful green text when running terraform plan. I applied the changes. Everything seemed to be running smoothly. The certificate validation was taking a while but that's normal, right? Why, though, was it taking so long? An hour? Two hours, 6 hours?? Something is wrong. What could it be? I double, triple checked my code. I rewrote the modules, tried different configurations for the creation of the the certificate and Route 53 records, read and read and read about similar problems people had when attempting DNS validation via Terraform (of which there were many). I tried every single thing I could find but, much to my shegrin, same result: Creating... Creating... Creating.... This was very discouraging, in all honesty. I had several video calls with my mentor and attempted to troubleshoot, and even he was stumped. At this point, I would estimate I had spent around 32 hours trying to solve this one single issue. I decided to take some space from the issue and work on refining my index.html document with the hopes that some clarity would come to me. After a while, a thought occured to me: What if it's my domain? It had been a while since my previous architecture was functional (seeing as how I tore everything down), and I realized that I had not recently verified that the hosted zone was even viable. I mean, why wouldn't it be, right?? I performed an ns dig and lo and behold: and error message. No.. it can't be. Turns out my email had sent the validation link for my purchased domain to my spam folder and as a result I failed to validate it, suspending the hosted zone. I validated and wouldn't you know: the certificate validated in about 45 seconds. This was met by a host of emotions: primarily relief mixed with a nice dash of incredulity and self-facing frustration. I spent SO much time pouring over and over and over the code adjusting every little detail, configuring and reconfiguring, double and triple checking all my credentials and variables and policies and roles just for it to end up being something as simple as my email sending something to a spam folder. I learned a valueable lesson that day: Investigate namespace early in the process. It takes only a moment to do and it checks off a large troubleshooting blind-spot.
In spite of the issue with ACM validation, working with Terraform was still my favorite part of the process of the Cloud Resume Challenge. It was really rewarding to orchestrate everything to work together and to organize it in a way that makes sense to me. Espescially after those, shall we say, "speed bumps" along the way, getting that successful Terrraform apply message felt fantastic. It is a tool that I will look forward to utilizing in my career moving forward.