Restore a Job with a Scoped Key

Introduction

The exposed support key has been revoked. The service is available, but the summary job can no longer authenticate because its private configuration still contains the old value. The pending daily batch still needs its summary.

You will learn scopes, issue a replacement, update the worker's private configuration, and complete a real business run. You will also test that customer-directory access stays forbidden and the unrelated monitor remains healthy. This lab supplies its own post-containment environment; it does not reuse the previous VM.

Test a Read-Only Credential

In this step, you will distinguish an invalid credential from a valid credential with insufficient permission. This fresh environment starts after containment: key_support_old is revoked, the job still uses its old secret, and the daily batch has not been processed. You do not need files or a VM from the previous lab. Keys and operator tokens belong to this VM; obtain the values here instead of reusing a previous environment's credentials.

Workstation is the maintainer terminal in /home/labex/project/repository. Integration runs as keyworker in /home/labex/project. Key Manager is the provider's maintenance UI. The local training API is at http://127.0.0.1:8000; its behavior is synthetic and specific to this course.

A scope is permission to perform a named operation. Least privilege means granting the operations needed for the job and leaving unrelated operations unavailable. The summary program reads tickets and publishes summaries; it does not read the customer directory.

Recognize a permission failure

A prepared read-only demonstration key is valid but lacks publishing permission. In Workstation, send it a publishing request. curl -X POST selects the write operation; -H supplies headers, -d supplies the JSON body, -s hides progress, and -w prints the status. The Content-Type: application/json header tells the service that the body uses JSON. The empty object {} is enough here because the provider rejects permission before checking the body:

curl -s -w '\nHTTP %{http_code}\n' -X POST -H 'Authorization: Bearer lab_demo_read_only' -H 'Content-Type: application/json' -d '{}' http://127.0.0.1:8000/v1/summaries

{"error":"Missing scope","required":"summaries:write"}
HTTP 403

For this API, 403 means the credential authenticated but lacks the required scope. Contrast it with 401, which rejects the credential itself. A read-only key cannot complete a read-plus-write job; an all-access key would allow more than necessary. Your replacement needs the two business scopes in the table.

Issue a Dedicated Replacement

In this step, you will issue a dedicated replacement with the two scopes selected in the previous step. You will save its secret privately in the next step.

A name is a useful label, but only the scopes control permission.

Issue a dedicated credential

In Workstation, print the operator token with cat, which displays a file:

cat /home/labex/.config/keylab/operator-token

In Key Manager, enter the value in Operator token, select Sign in, and follow Open Key Manager. Operator authentication is separate from application keys. Confirm that key_support_old is already Revoked.

Under Issue a key, enter Support summary replacement as Name. Select only tickets:read and summaries:write, then select Issue key. The UI displays a non-secret key ID and Secret shown once. Copy the secret value, beginning lab_key_, before leaving this page. The provider stores a hash rather than a recoverable secret. A hash is a one-way fingerprint: it lets the provider compare a supplied value without displaying the stored original.

If you lose this value, revoke that unused key by its ID and issue another. Do not reuse the exposed value or broaden permissions to troubleshoot a copying mistake.

Save the Private Runtime Configuration

In this step, you will save the replacement in the service account's private runtime file and check its access permissions. Use Integration for every command below.

A runtime configuration is the file the program reads while running. This worker reads /home/labex/project/runtime/job.ini on each invocation, so a restart is unnecessary. The repository's example file is not that file. In Integration, inspect the current configuration:

cat /home/labex/project/runtime/job.ini

[integration]
api_key = lab_key_support_exposed

Keep secrets out of the repository and shell command history. Bash read accepts a value at a prompt: -r preserves literal characters, -s hides what you type, and -p prints a prompt. API_KEY is a temporary shell variable. Run this command in Integration, paste the new secret at the prompt, then press Enter. The input is hidden:

read -r -s -p 'Replacement key: ' API_KEY

printf writes formatted text. Here \n makes line breaks, %s inserts the variable's value, double quotes preserve it as one argument, and > replaces the named file. The existing private file belongs to keyworker, the account in this terminal:

printf '[integration]\napi_key = %s\n' "$API_KEY" > /home/labex/project/runtime/job.ini

Use chmod to set access permissions. Mode 600 permits only the file owner to read and write; group members and other accounts get no access:

chmod 600 /home/labex/project/runtime/job.ini

ls -l shows ownership and permissions without printing the secret:

ls -l /home/labex/project/runtime/job.ini

Look for -rw------- and owner keyworker. The file size and date can vary. A shell variable alone does not update this file; saving the runtime file is what makes the next worker invocation use the new credential.

Keep this Integration terminal open. API_KEY belongs to this shell, not to every terminal. The next step uses it for independent requests, then removes it. If you close or reconnect the terminal, repeat this step using the current secret from job.ini, shown by the cat command above.

Complete and Verify the Business Run

In this step, you will process the pending daily batch and test both permitted and forbidden behavior. The worker is a prepared program: keep /opt/keylab/worker.py unchanged. The task is credential recovery, not changing the program to bypass authentication.

In Integration, run the assigned batch:

python3 /opt/keylab/worker.py daily

Read batch: daily (3 items)
{"batch":"daily","count":3,"subjects":["Reset password","Invoice copy","Delivery question"]}
Received batch: daily
Receiving key ID: ...

The receiving ID must match your replacement's ID in Key Manager. If you get 401, check that the saved secret is the active replacement. If you get 403, compare its scopes with the endpoint table. If necessary, revoke an incorrectly scoped replacement, issue a correctly scoped one, update the private file, and rerun the batch. Repeating this batch replaces its received record rather than creating duplicate summaries.

Inspect the actual receiving service

Stay in the same Integration terminal. The API_KEY variable from the previous step is still available here; you do not need to enter the one-time secret again.

Use the read-only results endpoint. A pipeline | passes the response to the next program; python3 -m json.tool runs Python's JSON formatter for readable indentation:

curl -s -H "Authorization: Bearer $API_KEY" http://127.0.0.1:8000/v1/results | python3 -m json.tool

Find batch daily, its three subjects inside payload, and your replacement's key_id. This record comes from the receiving API, not a local success file. The backend also records which credential read the source data.

Now test a forbidden customer-directory request:

curl -s -w '\nHTTP %{http_code}\n' -H "Authorization: Bearer $API_KEY" http://127.0.0.1:8000/v1/customers

{"error":"Missing scope","required":"customers:read"}
HTTP 403

This expected rejection confirms the new key is restricted. Use unset to remove the temporary shell variable when you no longer need it; the saved runtime file stays unchanged:

unset API_KEY

In Workstation, confirm that the old key remains unusable and the monitor remains healthy:

curl -s -w '\nHTTP %{http_code}\n' -H 'Authorization: Bearer lab_key_support_exposed' http://127.0.0.1:8000/v1/tickets

{"error":"Invalid or revoked key"}
HTTP 401

curl -s -w '\nHTTP %{http_code}\n' -H 'Authorization: Bearer lab_key_monitor_only' http://127.0.0.1:8000/v1/health

{"status":"ok","integration":"monitor"}
HTTP 200

Finally, git status --short lists changed or untracked repository files in a compact format:

git status --short

The guided path leaves no output because you edited only the private runtime file. Never put the replacement value in an example or commit. History can retain secrets after their visible lines disappear, as the first lab demonstrated.

Summary

You distinguished authentication failure from missing permission, selected a read-plus-write scope pair, saved a one-time secret privately, and completed the pending summary. You checked the receiving record, old-key rejection, forbidden customer access, and monitor availability. A useful rotation restores legitimate work and removes the exposed credential's access.