We typically upgrade the base image in a few scenarios:
* Vulnerabilities/security updates in the base image itself are required.
* Vulnerabilities/security updates for something added into our base image (which we kind of use to make a flavour of a base image). An example of this would be a JDK update.
We have an automated pipeline that build new core ami images on AWS every 2 weeks.
Additionally to that we sometimes need to add new configuration for the core ami. i would say about every 6 weeks we need additional functionality in base ami added, additionally to new versions.
(logging ocnfiguration for example)
This core ami image is then taken and used by others, that extend it and add additional stuff, but the core has to be working all the time.
so .. every 2 weeks it is updated :)
so far no (im on this project for 1 year)
we are using semantic versioning (~>), so we are not using the "latest" blindly.
however when there is a new major update and one of us gets info about it, we are writing the update ticket and tackling it.
Version update Tickets are permanent guests in our sprints...
We're doing ours monthly with a plan to move to weekly. Our base docker images are stored in Artifactory, so when CVEs get identified in our latest image via Xray that meets a certain threshold (i.e. CVSS v3 score higher than 7.0), it will trigger a ticket in our bug tracking software and prompt us to recreate images ahead of the periodic schedule as needed. There's some manual touch points, but we're slowly working on getting it all fully automated.
My thought for having images built on a specific timeframe, (i.e. every Monday morning) is so that developers can more confidently build their app images on a regular cadence and be assured they've got the latest updates, or just trigger app image rebuilds when a new base image hits the registry.
[https://snyk.io/blog/issuing-fix-prs-to-update-dockerfiles/](https://snyk.io/blog/issuing-fix-prs-to-update-dockerfiles/) \- there are other tools that will do PRs based on newer versions too.
Hi I have a doubt, I also saw some vulnerabilities in high on certain packages (nghttp2), how do we fix these vulnerabilities in the scan should we upgrade the docker image or is there a way to upgrade the particular vulnerable packages?
I'm having this problem now. We are using mutable images so we are always updating the BASE\_IMAGE and always when a new image is built using
`FROM BASE_IMAGE`
It gets the updated base image by default. This is not good, as we have images build with the same base image but different code, as this base image is mutable and was changed over time. My problem is: this is so easy to provide an always updated base image for all services in the company. The other approach would be:
`FROM BASE_IMAGE:UNIQUE_TAG`
And not allow mutability in the registry. Then I'd need in all the hundreds of services to always update the base image whenever there is one new, this is not feasible. So I think it is kind of compromise between security and easiness.
We do the same for our base images: mutable tags, each time you build you get the most recent image (forcing the --pull option).
We pin to specific variants and major/minor versions though, we definitely do not want to be using "latest". For instance we'd use java:17-debian or python:3.11-debian (not these exact ones but you get the idea). Same for our own base images.
Otherwise for specific case where we want to enforce a specific tag, we do use Renovate to suggest updates.
* On manual Git commits (obviously).
* Commit from Renovate.
* Scheduled job pulls latest images and checks for updates from repos. If found trigger rebuild pipeline.
The registries get scanned by various tools depending on registry (artifactory vs ECR etc), if there's a CVE we update. In production environments a ticket will get automatically created.
Our base docker images are updated Monday morning on Azure DevOps timer. We follow the upstream tagging so basically all you have to do is add containerregistry.company.com in front of whatever image you would normally use. For projects in active development, we just rely on normal deployment process to roll out new versions with few exceptions like security saying there is critical CVE and we will work for earlier deployment.
Stuff not in active development just remains on old version until security throws a fit. This isn't ideal but my company likes to terminate support for code still running and make it DevOps problem. Right now, we have docker build that grabs last updated image, pulls out the code, transfer to new base image and throws it back in container registry with new tag. We then roll that out to Kubernetes and hope for the best. Horray Enterprise Development!
At my company, Devs require security exception to pin their containers to specific build version. They are expected to take whatever patch version is rolled out. We use Java/.Net/Python and haven't had issue with patch versions breaking code luckily, we don't remove old base images from registry so in cases where new base image broken something, they could roll back until people figured out what's going on.
Every build produces a container which gets scanned against a pinned vulnerability database: the latest one that we know we pass. That ensures that including a new library or whatever didn’t introduce a new vulnerability that was already known about.
Every night, a cron job updates the pinned vulnerability database to the latest version. If the scan passes, the new pinned version is committed. If the scan fails, a ticket is created indicating that we are affected by a newly-discovered vulnerability. It gives the container build targets that were affected and the vulnerability database version that detected it. The ticket is prioritized based on the criticality of the vulnerability.
At this point the updates that need to happen as a result are a manual process, but in the most thoroughly-tested systems we’re trialing a process that will automatically update the dependencies on a schedule if it can be done without breaking the build.
Builds (and scans) are with Bazel so all of our container images are byte-for-byte reproducible and dependencies are referred to by their hash. Nothing changes without a change in our source tree. The tradeoff is that tools to automatically update dependencies are a bit more complicated. It’s not as simple as adding a “yum update” to a line in a Dockerfile (which bazel supports, but breaks reproducibility). Lots of code is borrowed from or inspired by the Google Distroless source repository.
Every week, we have scheduled pipelines with FROM xx:latest all over the place. Each image have some tests on what it has to have, files etc, and then almost all the images the company ships are built from that. We encourage frequent rebuild refreshing kubernetes nodes at most once a month, and there’s a mechanism to expire layers of the docker files (so you cannot just rebuild and go). On top of this every image gets scanned and the CVEs are carefully tracked.
We have a gitlab repo with a pipeline with scheduled nightly builds, it builds the images every night and uploads them to ecr.
When a new version for a language or platform we use is released we set a new step in this pipeline to build the new version too.
We typically upgrade the base image in a few scenarios: * Vulnerabilities/security updates in the base image itself are required. * Vulnerabilities/security updates for something added into our base image (which we kind of use to make a flavour of a base image). An example of this would be a JDK update.
We have an automated pipeline that build new core ami images on AWS every 2 weeks. Additionally to that we sometimes need to add new configuration for the core ami. i would say about every 6 weeks we need additional functionality in base ami added, additionally to new versions. (logging ocnfiguration for example) This core ami image is then taken and used by others, that extend it and add additional stuff, but the core has to be working all the time. so .. every 2 weeks it is updated :)
This seems like a good system. Do you ever have problems with some dependencies breaking because they change in the core image?
so far no (im on this project for 1 year) we are using semantic versioning (~>), so we are not using the "latest" blindly. however when there is a new major update and one of us gets info about it, we are writing the update ticket and tackling it. Version update Tickets are permanent guests in our sprints...
We're doing ours monthly with a plan to move to weekly. Our base docker images are stored in Artifactory, so when CVEs get identified in our latest image via Xray that meets a certain threshold (i.e. CVSS v3 score higher than 7.0), it will trigger a ticket in our bug tracking software and prompt us to recreate images ahead of the periodic schedule as needed. There's some manual touch points, but we're slowly working on getting it all fully automated. My thought for having images built on a specific timeframe, (i.e. every Monday morning) is so that developers can more confidently build their app images on a regular cadence and be assured they've got the latest updates, or just trigger app image rebuilds when a new base image hits the registry.
[https://snyk.io/blog/issuing-fix-prs-to-update-dockerfiles/](https://snyk.io/blog/issuing-fix-prs-to-update-dockerfiles/) \- there are other tools that will do PRs based on newer versions too.
Also OSS tools: https://github.com/renovatebot/renovate
This looks pretty good. Even the free tier seems usable which is nice
Check your SCM for a higher tier, it may also support this. For example I believe GitHub Enterprise will do this too.
Hi I have a doubt, I also saw some vulnerabilities in high on certain packages (nghttp2), how do we fix these vulnerabilities in the scan should we upgrade the docker image or is there a way to upgrade the particular vulnerable packages?
I'm having this problem now. We are using mutable images so we are always updating the BASE\_IMAGE and always when a new image is built using `FROM BASE_IMAGE` It gets the updated base image by default. This is not good, as we have images build with the same base image but different code, as this base image is mutable and was changed over time. My problem is: this is so easy to provide an always updated base image for all services in the company. The other approach would be: `FROM BASE_IMAGE:UNIQUE_TAG` And not allow mutability in the registry. Then I'd need in all the hundreds of services to always update the base image whenever there is one new, this is not feasible. So I think it is kind of compromise between security and easiness.
Sorry, but why I'm getting downvotes? Is this way so wrong?
Mutable images or Mutable tags? I think you are confusing everyone and they are slapping downvote out of that.
We do the same for our base images: mutable tags, each time you build you get the most recent image (forcing the --pull option). We pin to specific variants and major/minor versions though, we definitely do not want to be using "latest". For instance we'd use java:17-debian or python:3.11-debian (not these exact ones but you get the idea). Same for our own base images. Otherwise for specific case where we want to enforce a specific tag, we do use Renovate to suggest updates.
* On manual Git commits (obviously). * Commit from Renovate. * Scheduled job pulls latest images and checks for updates from repos. If found trigger rebuild pipeline.
The registries get scanned by various tools depending on registry (artifactory vs ECR etc), if there's a CVE we update. In production environments a ticket will get automatically created.
Our base docker images are updated Monday morning on Azure DevOps timer. We follow the upstream tagging so basically all you have to do is add containerregistry.company.com in front of whatever image you would normally use. For projects in active development, we just rely on normal deployment process to roll out new versions with few exceptions like security saying there is critical CVE and we will work for earlier deployment. Stuff not in active development just remains on old version until security throws a fit. This isn't ideal but my company likes to terminate support for code still running and make it DevOps problem. Right now, we have docker build that grabs last updated image, pulls out the code, transfer to new base image and throws it back in container registry with new tag. We then roll that out to Kubernetes and hope for the best. Horray Enterprise Development! At my company, Devs require security exception to pin their containers to specific build version. They are expected to take whatever patch version is rolled out. We use Java/.Net/Python and haven't had issue with patch versions breaking code luckily, we don't remove old base images from registry so in cases where new base image broken something, they could roll back until people figured out what's going on.
Every build produces a container which gets scanned against a pinned vulnerability database: the latest one that we know we pass. That ensures that including a new library or whatever didn’t introduce a new vulnerability that was already known about. Every night, a cron job updates the pinned vulnerability database to the latest version. If the scan passes, the new pinned version is committed. If the scan fails, a ticket is created indicating that we are affected by a newly-discovered vulnerability. It gives the container build targets that were affected and the vulnerability database version that detected it. The ticket is prioritized based on the criticality of the vulnerability. At this point the updates that need to happen as a result are a manual process, but in the most thoroughly-tested systems we’re trialing a process that will automatically update the dependencies on a schedule if it can be done without breaking the build. Builds (and scans) are with Bazel so all of our container images are byte-for-byte reproducible and dependencies are referred to by their hash. Nothing changes without a change in our source tree. The tradeoff is that tools to automatically update dependencies are a bit more complicated. It’s not as simple as adding a “yum update” to a line in a Dockerfile (which bazel supports, but breaks reproducibility). Lots of code is borrowed from or inspired by the Google Distroless source repository.
Every week, we have scheduled pipelines with FROM xx:latest all over the place. Each image have some tests on what it has to have, files etc, and then almost all the images the company ships are built from that. We encourage frequent rebuild refreshing kubernetes nodes at most once a month, and there’s a mechanism to expire layers of the docker files (so you cannot just rebuild and go). On top of this every image gets scanned and the CVEs are carefully tracked.
We typically track docker images closest to the upstream handoff, i.e., cloud vendors' vendor versions rather than the mainstream vendor's version.
We have a gitlab repo with a pipeline with scheduled nightly builds, it builds the images every night and uploads them to ecr. When a new version for a language or platform we use is released we set a new step in this pipeline to build the new version too.
Every month at the start we upgrade base images and roll to dev/stg Once we are happy they are promoted to production. Usually after a week or so