DevOps NetworkDevOpsNetwork
Join Free
Practical infra coding

Deep-merge layered config files

mediumCSV, JSON and YAML

Problem statement

Services are configured in layers: a base.json shared by every environment, then an override per environment. Merge any number of layers, left to right, with these rules:

  • Objects merge key by key, recursively.
  • Lists and scalars in a later layer replace the earlier value entirely.
  • A null in a later layer deletes the key.
  • The input objects must not be modified. The caller may reuse them.
  • If a key changes between object and non-object, the later layer wins, with a warning on stderr.

Print the result as JSON with 2-space indentation.

base.json

JSON
{
"replicas": 2,
"image": {"repo": "shop/api", "tag": "1.4.0"},
"resources": {"cpu": "250m", "memory": "256Mi"},
"env": {"LOG_LEVEL": "info", "FEATURE_X": "off"},
"ports": [8080, 9090]
}

prod.json

JSON
{
"replicas": 6,
"image": {"tag": "1.4.2"},
"resources": {"memory": "1Gi"},
"env": {"FEATURE_X": null, "REGION": "eu-west-1"},
"ports": [8080]
}

Examples

Example 1

Input: python solution.py base.json prod.json

Output: { "replicas": 6, "image": { "repo": "shop/api", "tag": "1.4.2" }, "resources": { "cpu": "250m", "memory": "1Gi" }, "env": { "LOG_LEVEL": "info", "REGION": "eu-west-1" }, "ports": [ 8080 ] }

Explanation: image.repo survives because objects merge. ports is replaced, not appended. FEATURE_X: null removes the key.

Hints

Approach

Optimal

A recursive merge that returns a new structure.

  1. copy.deepcopy(base) makes the result independent of the input, so later mutations cannot leak back into a cached base config.
  2. For each key in the override: None pops the key. A type change between object and non-object prints a warning.
  3. A dict value recurses into the existing dict, or into {} if there is none. Merging onto {} matters: it makes {"new": {"a": null}} produce {"new": {}} instead of storing a literal null.
  4. Any other value is deep-copied in, so lists are replaced whole.

main folds the layers left to right. Each layer is visited once, and the copies make the cost proportional to the total size of the documents.

ComplexityTime O(total size)Space O(total size)
Python
import copy
import json
import sys
def deep_merge(base, override, path=""):
"""Return a new dict: override wins, dicts merge recursively,
lists and scalars are replaced whole, and null deletes the key."""
result = copy.deepcopy(base) # never mutate the caller's data
for key, value in override.items():
where = f"{path}.{key}" if path else key
if value is None:
result.pop(key, None)
continue
existing = result.get(key)
if key in result and isinstance(existing, dict) != isinstance(value, dict):
print(f"warning: {where} changes type "
f"({type(existing).__name__} -> {type(value).__name__})", file=sys.stderr)
if isinstance(value, dict):
# merge onto {} when there is nothing to merge with, so nested nulls still delete
result[key] = deep_merge(existing if isinstance(existing, dict) else {}, value, where)
else:
result[key] = copy.deepcopy(value)
return result
def load(path):
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
except json.JSONDecodeError as e:
sys.exit(f"{path}:{e.lineno}:{e.colno}: invalid JSON: {e.msg}")
if not isinstance(data, dict):
sys.exit(f"{path}: top level must be an object")
return data
def main(paths):
merged = {}
for p in paths: # later layers win: base, then env, then region, ...
merged = deep_merge(merged, load(p))
print(json.dumps(merged, indent=2))
if __name__ == "__main__":
main(sys.argv[1:] or ["base.json", "prod.json"])

Follow-up questions

  • Merge lists of objects by their name field, like Kubernetes containers.
  • Report where each final value came from (replicas: prod.json), for debugging.
  • Support environment-variable overrides such as APP__DATABASE__HOST=... as the last layer.

Frequently asked questions

There is no single right way to merge lists: append, merge by index, or merge by a key such as name. Replacing is predictable and is what most tools do by default, including Helm values. Kustomize strategic merge patches merge by key for some fields, which is powerful but surprises people; that is a good follow-up to discuss.

Layered configuration is how Helm values, Kustomize overlays, Spring profiles and most deploy tools work. Bugs here, such as mutating the shared base or a null that fails to delete, cause environment-specific incidents that are hard to track down. The task tests recursion and care about aliasing.

If deep_merge changed base in place, merging prod would change the base, and a later staging merge in the same process would inherit prod's values. That bug appears only when several environments are rendered in one run, which makes it very hard to find.