HourTab

Blog › ICP guides

Infrastructure engineer on retainer: tracking platform infrastructure advisory and demonstrating operations value between Terraform module releases and Kubernetes cluster upgrades

July 23, 2026 · ~18 min read

The Terraform module library release and the Kubernetes cluster upgrade are the visible infrastructure engineering milestones. When an engineering director presents the platform team's quarterly work to the CTO, when a VP Platform reviews the infrastructure cost report, when an engineering manager presents the reliability summary to the product team — those are the artifacts on the table: the Terraform VPC and EKS module library that reduced environment provisioning from a three-day manual process to a forty-minute pipeline run with consistent configuration across development, staging, and production; the Kubernetes cluster upgrade from 1.28 to 1.29 that completed with zero service disruption because the node pool replacement strategy kept the old node group available while the new node group passed readiness checks; the Reserved Instance coverage increase from 12% to 67% of EC2 spend that reduced the monthly cloud bill by $4,200. What none of those artifacts shows is the continuous platform infrastructure advisory between those visible milestones, or whether that ongoing advisory identified the four Terraform modules had no version pinning and were using local relative path sources before a development environment module change propagated to production, calibrated the payment processor pod's memory limit to survive the monthly billing cycle's processing spike before the OOMKilled events accumulated into a customer-visible payment timeout incident, or recognized that fourteen gp2 EBS volumes on throughput-constrained instances could be migrated to gp3 at 31% lower cost before the quarterly infrastructure budget review.

The Terraform module versioning governance review that found four modules — vpc, eks, rds, and alb — were sourced using relative path references with no version tag pinning — where all three environments were consuming the same module source directory, meaning a change to the EKS module scoped to add a spot instance node group for development workloads would take effect in staging and production on the next terraform apply without an environment-specific promotion gate, pull request review targeting the production configuration, or change advisory board entry — where converting the module sources to versioned git references with semantic version tags, a module versioning policy requiring testing in development before the version reference was updated in staging, and a separate promotion review before updating the production environment configuration addressed the uncontrolled change propagation risk that the team did not know existed because the previous three months had happened to contain no module changes where the development scope differed from what was appropriate for production.

The Kubernetes resource calibration advisory that identified the payment-processor deployment was configured with a 256Mi memory request and a 512Mi memory limit — where the pod scheduler was placing the pod on nodes that had 512Mi of available memory, which was sufficient to satisfy the request but not the peak utilization — where production Prometheus metrics showed the p95 memory utilization during the monthly billing cycle's batch processing window reaching 840Mi, exceeding the 512Mi limit and generating OOMKilled events — where the OOMKilled events were surfacing in the application monitoring dashboard as intermittent HTTP 504 Gateway Timeout errors from the payment service rather than as Kubernetes pod eviction events in the platform team's alerting, because the service mesh was routing requests to the pod during the brief window between the OOMKill and the pod restart completing — where updating the memory request to 512Mi and the limit to 1Gi, and adding a Vertical Pod Autoscaler recommendation mode configuration to generate ongoing calibration recommendations as payment volume grew, prevented the monthly billing cycle processing spike from triggering a customer-visible payment failure before the billing team escalated it as a platform reliability incident.

The infrastructure cost optimization review that found three r5.2xlarge instances running the analytics query processing service at 7% average CPU utilization and 11% memory utilization over the trailing 90 days — where the instances had been provisioned at r5.2xlarge during an initial load test that used a query workload three times larger than the actual production query volume — where right-sizing to r5.large instances with four instances in the auto scaling group for availability (one more instance than the over-provisioned three-instance r5.2xlarge configuration) reduced the On-Demand cost of those instances from $1,847/mo to $924/mo — where the Reserved Instance coverage analysis identified six additional instances across the production account with stable 90-day utilization patterns where converting from On-Demand to one-year Reserved Instance pricing would reduce the per-instance cost by 34%, projecting $2,400/mo in additional savings beyond the right-sizing reduction — where the complete infrastructure cost advisory delivered $3,300/mo in sustained cost reduction before the quarterly budget review where the VP Platform was asked to explain why cloud spend was increasing despite stable product traffic.

Infrastructure engineers and platform infrastructure consultants on monthly retainer do their most consequential work in the continuous stretches between Terraform module releases and Kubernetes cluster upgrade milestones: the Terraform module governance that prevents development environment module changes from propagating to production through unversioned source paths, enforces provider version pinning to prevent provider API drift between environments, and maintains state backend configuration consistency to ensure every environment's state is stored remotely with locking; the Kubernetes cluster operations advisory that calibrates resource requests and limits against actual production utilization metrics rather than initial estimates, designs PodDisruptionBudgets to maintain service availability during cluster maintenance and node pool upgrades, and advises on the cluster upgrade sequencing that avoids simultaneous control plane and node pool upgrades; the infrastructure cost optimization advisory that identifies right-sizing opportunities from CloudWatch and Prometheus utilization metrics before over-provisioned instances accumulate into a significant budget variance, analyzes Reserved Instance and Savings Plan coverage for compute spend that has demonstrated the stability required for committed pricing, and identifies EBS volume types that are optimized for the wrong performance dimension for the workload they serve; and the CI/CD pipeline infrastructure advisory that sizes GitHub Actions self-hosted runners for the actual build and test workload's memory and CPU profile, configures artifact storage and Docker layer cache to eliminate redundant image build work, and designs the deployment pipeline stage structure with environment protection rules that prevent a failed stage gate condition from routing the wrong artifact to the production environment. All of that advisory is invisible to the engineering director and VP Platform without a work log that connects the ongoing infrastructure operations advisory to the platform reliability, cost, and velocity metrics it enables.

Terraform module governance advisory

Infrastructure-as-code consistency depends on the module versioning and state management practices that are invisible when they are working correctly and catastrophic in their failure modes when they are not. A Terraform configuration where the development, staging, and production environments source their VPC, EKS, and RDS modules from the same unversioned local directory has no mechanism to prevent a module change that is appropriate for development from reaching the production environment on the next apply cycle. A state backend where some modules use remote S3 state with DynamoDB locking and others use local state has no mechanism to prevent two engineers from applying conflicting changes to the same module simultaneously.

The module versioning failure mode that most commonly precedes a production infrastructure incident is not a misconfigured module parameter — it is the absence of a version reference that would cause the staging environment to test the module change before the production environment applied it. Teams that adopt Terraform for the first time typically use local module sources because the immediate priority is getting the infrastructure defined in code, and the versioning governance that would enforce environment promotion sequence is added later, after the first incident where a module change intended for development reached production because nothing in the workflow required the change to be promoted through staging first.

Terraform module governance advisory on retainer covers: reviewing the module source references across all environment configurations to identify modules that are using local path references without version pinning; advising on the semantic versioning convention and git tagging workflow that makes module versions promotable across environments independently; reviewing the state backend configuration to confirm all modules are using remote state with locking and that the state backend access credentials are scoped to prevent unauthorized state modification; advising on the workspace design for multi-environment module promotion, including the pull request workflow that requires a module version to be tested in development and staging before the production environment configuration is updated; reviewing the provider version constraints to confirm they are pinned to prevent provider API drift between environments; and reviewing the Terraform variable validation rules and output documentation to ensure module consumers have the information they need to configure the module correctly without reading the module implementation.

On retainer: monthly state consistency review confirming no drift between Terraform state and actual cloud resource configuration; quarterly module versioning audit confirming all environment configurations reference specific module versions rather than floating sources; and review of any module change pull request where the change scope differs between what is appropriate for development and what should be tested before reaching production.

Kubernetes cluster operations advisory

Kubernetes resource configuration errors rarely surface as Kubernetes events in platform team alerting. A pod with a memory limit set below its peak production utilization generates OOMKilled events that appear in the pod event log but typically surface to the application team as intermittent HTTP 503 or 504 responses in their service monitoring, not as an infrastructure configuration problem in the platform engineering backlog. A deployment with no PodDisruptionBudget configured and two replicas co-scheduled on the same node generates a service outage during the cluster maintenance window where the node pool is upgraded and the drain sequence removes both replicas simultaneously — surfacing as an unexpected outage during a planned maintenance window that the application team and the platform team had both expected to be non-disruptive.

Resource request calibration is the Kubernetes operations advisory function with the most consistent gap between the configured values and the actual production utilization. Teams that deploy new services typically set resource requests and limits based on development or load test environment observations, where the workload pattern differs from production in ways that are not visible until the service encounters its first production traffic spike, batch processing cycle, or concurrent user load peak. The memory request determines where the pod is scheduled; the memory limit determines the ceiling the container runtime enforces before triggering an OOMKill. A memory limit set at twice the development-observed utilization will be adequate until the first production condition that drives utilization above that ceiling — which is the condition the limit was intended to protect the node from, not to prevent the pod from surviving.

Kubernetes cluster operations advisory on retainer covers: reviewing pod resource request and limit configurations against actual CPU and memory utilization metrics from the cluster's monitoring stack, identifying pods where the configured limit is below the observed p95 utilization in production; advising on the node pool autoscaling configuration, including the scaling metric selection for the workload type (CPU-based HPA for compute-bound services, memory-based HPA for memory-bound services, custom metrics-based HPA for I/O-bound or queue-depth-driven services); reviewing PodDisruptionBudget configurations for all stateless deployments to confirm service availability is maintained during planned cluster maintenance; advising on the cluster upgrade sequencing, including the order in which control plane and node pool upgrades should be performed and the node pool replacement strategy that avoids simultaneous draining of all nodes running a given service; reviewing the cluster's node condition health to identify nodes with memory pressure or disk pressure conditions before they affect pod scheduling; and advising on the namespace resource quota and LimitRange configuration that prevents a single namespace from consuming disproportionate cluster capacity.

On retainer: monthly resource utilization review comparing configured requests and limits against the trailing 30-day p95 utilization for all production workloads; quarterly PodDisruptionBudget audit confirming all critical services have disruption budgets that match their availability requirements; and review of resource configuration for any new service before it is deployed to production.

Infrastructure cost optimization advisory

Cloud infrastructure cost growth follows a predictable pattern: initial deployments are provisioned with generous capacity estimates to avoid the performance problems that come with under-provisioning, and those capacity estimates are rarely revisited once the infrastructure is stable because the performance problems stopped occurring and the team moved on to building the next service. The cost of over-provisioned infrastructure accumulates steadily in the monthly cloud bill while the resource utilization data that would identify the right-sizing opportunity sits in CloudWatch or Prometheus, unreviewed because the cost growth appears linear and unalarming until the quarterly budget review where the total is compared against the prior year and the variance is difficult to explain with the available data.

Reserved Instance and Savings Plan coverage is the infrastructure cost optimization lever with the largest single-period savings potential and the most consistent under-utilization. Teams that are growing their infrastructure footprint typically maintain high On-Demand coverage because committing to Reserved Instance pricing requires confidence that the instance type and region will remain stable for one to three years. That confidence typically exists for the core database and application server infrastructure within six months of the initial deployment, but the Reserved Instance purchase workflow requires someone to do the coverage analysis, identify the stable instance populations, and execute the purchase — and that workflow typically falls outside the day-to-day operational priorities of the engineering team responsible for the infrastructure.

Infrastructure cost optimization advisory on retainer covers: monthly AWS Cost Explorer or cloud provider cost review to identify cost growth categories and anomalous spend patterns before they accumulate into a budget variance; right-sizing analysis for EC2 instances, RDS instances, and other compute resources using the trailing 90-day utilization metrics to identify instances where the provisioned size exceeds the utilization pattern; Reserved Instance and Savings Plan coverage analysis to identify instance populations with stable utilization patterns that qualify for committed pricing, and recommendation of the purchase amount and term that balances commitment risk against savings opportunity; EBS volume type analysis to identify volumes where the provisioned IOPS or throughput class is mismatched to the actual workload's performance requirements; data transfer cost analysis to identify cross-region and egress transfer patterns that could be reduced by caching, CDN configuration, or architecture changes; and quarterly infrastructure cost report preparation summarizing the right-sizing actions taken, the Reserved Instance coverage progress, and the projected savings trajectory.

On retainer: monthly cost review with anomaly identification; quarterly right-sizing sweep across all production compute resources; and Reserved Instance coverage analysis and purchase recommendation every six months as the committed pricing terms approach renewal.

CI/CD pipeline infrastructure advisory

CI/CD pipeline infrastructure failures are the platform engineering problems that software development teams experience most directly and attribute most readily to the platform team when they occur: the build that runs out of memory during the parallel test phase and produces a cryptic out-of-memory kill signal instead of a test failure output; the Docker image build that takes eleven minutes because the layer cache is not configured and every push rebuilds the full image from the base layer; the deployment pipeline that routes a staging artifact to the production environment because the environment selection logic has a condition that evaluates differently than the pipeline author expected. All three are infrastructure configuration problems, not application code problems, and all three generate engineer frustration that accumulates in the team's perception of the platform team's reliability.

GitHub Actions self-hosted runner sizing is the CI/CD infrastructure configuration failure with the most consistent impact on build reliability. Teams that provision self-hosted runners typically estimate the runner size based on the build's typical memory usage, not the peak memory usage during the parallel test execution phase that consumes significantly more memory than the build compilation and linting phases. A Maven build configured to run eight parallel test workers on a runner with 4GB of RAM will reliably produce OOMKilled build failures when the parallel test initialization phase loads eight JVM instances simultaneously, which is not the memory profile visible during the compilation phase that the initial runner size estimate was based on.

CI/CD pipeline infrastructure advisory on retainer covers: reviewing the self-hosted runner configuration for memory and CPU sizing against the actual peak utilization during the build and test phases, using runner metrics to identify which build phase is resource-constrained; advising on the Docker image layer cache configuration using the GitHub Actions cache action or a dedicated cache backend, including the cache key strategy that maximizes cache hit rate across parallel job runs on the same branch; reviewing the deployment pipeline stage configuration for environment protection rules, artifact provenance tracking, and deployment approval gates that prevent the wrong artifact from reaching the production environment; advising on the artifact storage and retention configuration to balance artifact availability for rollback against storage cost accumulation; reviewing the pipeline secret management configuration to confirm secrets are scoped to the environments that require them and are rotated on a schedule consistent with the organization's security policy; and advising on the runner fleet autoscaling configuration for workloads with variable build concurrency, including the scale-to-zero configuration for repositories with infrequent build activity.

On retainer: monthly pipeline health review covering build success rate, mean build duration trend, and runner resource utilization; quarterly artifact storage audit to confirm retention policies are removing aged artifacts that are no longer candidates for production deployment; and review of the pipeline configuration for any new service repository before its first production deployment.

The work that most commonly goes unlogged in an infrastructure engineering retainer

The most consistently underlogged infrastructure engineering advisory falls into two patterns: review sessions that confirmed the existing configuration was correct and required no adjustment, and advisory work that prevented an infrastructure failure from reaching the production availability incident threshold rather than contributing to a visible reliability win. Both patterns produce the misimpression that the retainer period was light when it contained the continuous infrastructure governance that enables the platform reliability and cost efficiency that engineering leadership experiences as the infrastructure simply working as expected.

Terraform state consistency review sessions where no drift was detected between the state and the actual infrastructure are the canonical underlogging case. Reviewing the EKS module state across all three environments to confirm the node pool configuration matched the pinned module version, the state backend was accessible with the correct credentials, and no manual console changes had introduced drift — that required the same state audit process as the session that identified the drift from an engineer who had modified a security group rule directly in the AWS console during an incident response and had not reversed the change after the incident was resolved. The engineering director who knows the Terraform state was audited and confirmed consistent before the quarterly compliance review is in a materially different position than one who assumed consistency without the audit that established it.

Kubernetes resource review sessions where all monitored workloads were within calibrated limits with no OOMKilled events in the trailing 30 days are consistently underlogged by infrastructure engineers who conflate “no finding” with “no advisory was performed.” Reviewing the Prometheus utilization metrics for all production deployments, confirming the p95 memory utilization was below 70% of the configured limit for every service, and confirming no pod had generated an OOMKilled event in the past 30 days — that required the same utilization analysis as the review that identified the payment processor pod's memory limit would be exceeded during the monthly billing cycle processing spike. The stable service availability that the application team experiences during the billing cycle processing window is not a coincidence; it is the outcome of the resource calibration review that identified and corrected the misconfiguration before it generated a customer-visible payment failure.

Retainer rates for infrastructure engineers and platform infrastructure consultants

Infrastructure engineer and platform infrastructure consultant retainer rates vary with cloud platform depth, infrastructure-as-code maturity, and Kubernetes expertise:

Advisory-only retainers covering Terraform governance review, Kubernetes operations advisory, cost optimization analysis, and CI/CD pipeline architecture are priced differently from retainers that include execution work such as applying Terraform changes, executing cluster upgrades, or modifying production pipeline configuration directly. The advisory function that governs the infrastructure configuration standards and prevents misconfiguration from becoming a production incident or a budget variance is the ongoing retainer function; the execution work that implements those standards is separately scoped.

Making infrastructure engineer retainer advisory visible to engineering leadership

The central challenge in infrastructure engineering retainer relationships is that the value of ongoing platform infrastructure advisory is structurally invisible to engineering directors and VPs Platform when the advisory is working as intended: the Terraform module library that enables consistent environment provisioning does not show the versioning governance review that prevented the development module change from propagating to production; the Kubernetes cluster that completed its upgrade with zero service disruption does not show the PodDisruptionBudget advisory that prevented simultaneous pod draining on the node being replaced; the monthly cloud bill that stayed within the infrastructure budget does not show the right-sizing and Reserved Instance coverage analysis that absorbed the new service's compute cost within the existing budget envelope.

The work log that connects advisory sessions to specific infrastructure configuration decisions, cost optimization recommendations, and operational governance reviews is the primary mechanism for making infrastructure engineering advisory value visible over time. An entry that records the Terraform state audit, the module versioning analysis, and the pinning recommendation that closed the uncontrolled change propagation risk gives the engineering director a concrete example of what the infrastructure governance retainer prevents. An entry that records the Kubernetes resource calibration review, the Prometheus utilization analysis, and the limit correction that prevented the monthly billing cycle OOMKill demonstrates the kind of continuous operations advisory that keeps the platform reliability statistics stable rather than accumulating incidents that surface at the quarterly review.

A retainer dashboard that makes the infrastructure engineer's work log visible to the engineering director or VP Platform without requiring a separate monthly infrastructure review converts the work log from a private platform team record into a shared infrastructure governance artifact. The engineering leader who can see the full quarter's Terraform governance reviews, Kubernetes resource calibration sessions, cost optimization analyses, and pipeline infrastructure advisory in a single URL understands immediately what the infrastructure engineering retainer is producing — and has a concrete record to reference when planning the platform team budget, evaluating whether the advisory retainer scope matches the infrastructure's growth trajectory, or explaining to the CFO why the cloud infrastructure cost trend is stable despite a 40% increase in production service count.

Frequently asked questions

What does an infrastructure engineer on retainer typically do?

An infrastructure engineer or platform infrastructure consultant on monthly retainer provides Terraform module governance (module versioning with semantic version tags and environment promotion gates, state backend configuration with remote locking, provider version pinning, variable validation), Kubernetes cluster operations advisory (resource request and limit calibration against production utilization metrics, node pool autoscaling configuration, PodDisruptionBudget design, cluster upgrade sequencing), infrastructure cost optimization advisory (right-sizing from CloudWatch utilization metrics, Reserved Instance and Savings Plan coverage analysis, EBS volume type optimization, data transfer cost reduction), and CI/CD pipeline infrastructure advisory (runner sizing for peak build memory profile, Docker layer cache configuration, deployment pipeline stage design with environment protection rules). The Terraform module release and the Kubernetes cluster upgrade are the visible milestones; the continuous infrastructure operations advisory that prevents module drift, resource exhaustion, cost overrun, and pipeline failures between those milestones is the ongoing retainer function.

What infrastructure engineer retainer work is most commonly underlogged?

Terraform state review sessions where state was confirmed consistent with no drift detected, Kubernetes resource review sessions where utilization was within calibrated limits and no OOMKilled events occurred in the trailing 30 days, infrastructure cost monitoring sessions where spend was confirmed within the established baseline with no anomalous growth categories, CI/CD pipeline health reviews where the build success rate was stable and no runner capacity issues appeared, and node group health reviews where all node pools were at expected capacity with no degraded conditions. All represent genuine infrastructure governance whose value is in the ongoing monitoring and problem prevention rather than in a visible incident resolution.

What should an infrastructure engineer retainer agreement include?

Environment access scope (which cloud accounts, Kubernetes clusters, Terraform state backends, and CI/CD systems the consultant can review), infrastructure repository access requirements (Terraform module repositories, Kubernetes manifest repositories, CI/CD pipeline configuration files), scope boundary between advisory and execution (advisory retainer covers governance review, resource calibration advisory, cost analysis, and pipeline architecture advisory; applying Terraform changes, executing cluster upgrades, and modifying production pipeline configuration directly are separately scoped), change management requirements for how advisory outputs are implemented, and a shared work log visible to the engineering director documenting the infrastructure governance reviews and operational advisory the retainer produces.

What are typical retainer rates for infrastructure engineers?

Mid-level infrastructure engineers (3–6 years, Terraform proficiency, AWS or GCP certification, Kubernetes administrator): $110–$175/hr, typically $1,100–$3,500/mo. Senior infrastructure engineers or platform infrastructure leads (6–12 years, multi-cloud Terraform, cluster lifecycle management, FinOps, CI/CD infrastructure design): $160–$260/hr, typically $2,400–$6,500/mo. Principal infrastructure engineers or Staff platform engineers (12+ years, large-scale distributed infrastructure, multi-region Kubernetes, enterprise FinOps program governance): $210–$380/hr, typically $4,200–$13,300/mo.

How should infrastructure engineer retainer hours be logged?

Log entries should capture the infrastructure domain (Terraform governance, Kubernetes operations, cost optimization, CI/CD pipeline infrastructure), the specific resource or environment context, the advisory work performed, and the recommendation or finding. Log every advisory session, including those that confirmed existing configuration was correctly sized and required no adjustment — the Terraform state review that confirmed no drift required the same audit process as the review that identified the console-modified security group rule that had not been reverted after the incident, and the absence of a finding is the outcome of the monitoring that would have found the finding if it had been there.

HourTab turns a time-tracker CSV into a public retainer-hours URL your client can bookmark. No client login required. See how it works →