Assignment #6 - Usable security

In this assignment, you will first break provided CAPTCHA images, then design an improved CAPTCHA with respect to usable security (hard for bots, readable for humans), test the improved CAPTCHA design with the same solver and evaluate both CAPTCHA designs.

You need to implement an Optical Character Recognition (OCR) solution that processes CAPTCHA images, extracts the text using OCR, and evaluates the accuracy by comparing the extracted text from the CAPTCHA.

You are given the following for the homework:

• solver_template.zip that contains:
  • solver.py: A starting Python script to implement the CAPTCHA-solving solution.
  • requirements.txt: A list of required Python packages for the project.
  • HW06_Report_Template.docx: A Word document template for your final report.
  • captchas.zip: A ZIP of 20 CAPTCHA images to process and solve.

Detailed Instructions

1. Implement a CAPTCHA solver using OCR by completing solver.py. The following parts need to be implemented inside the template:
   1. Importing the required libraries
   2. Defining a function to solve a CAPTCHA image
   3. Setting the path to the CAPTCHA folder
   4. Solve 20 captchas (read text in png image using OCR) and evaluate
   5. Calculating the OCR accuracy by comparing filenames to extracted text
   6. Saving the results to a JSON file


2. Use your solver to try to bypass the provided CAPTCHA images and calculate the OCR accuracy by comparing extracted text to the filenames. (expected is at least 50% accuracy).


3. Edit 20 CAPTCHA images to make them harder for OCR to solve, while keeping them readable for humans (decrease OCR accuracy). Possible improvements include adding noise, distortion, overlapping characters, varying fonts, or obfuscation layers. For improving the CAPTCHA design, you are free to use any techniques or tools.


4. Use the same solver to try to bypass the improved CAPTCHA images and measure the new accuracy. (expected is lower accuracy than in step 2.).


5. Write a report using the provided HW Report Template that includes:
   1. The OCR accuracy on the original CAPTCHAs
   2. The OCR accuracy on the improved CAPTCHAs
   3. An evaluation comparing both CAPTCHA designs from a usable security perspective (human usability vs. OCR resistance)

Note

The solver solver.py generates a results.json file containing the accuracy evaluation of the OCR on CAPTCHA images.

Deliverables

• Python script solver.py with your completed implementation
• Two output files results_original.json and results_improved.json containing OCR results for the original and improved CAPTCHAs
• 20 improved CAPTCHA images (in PNG format) zipped together
• Report in PDF edited and exported from the provided HW Report Template

Scoring Rubric (Total: 6 points)

CAPTCHA Solving and Accuracy Evaluation (2 points)

• 2 points: Fully implemented OCR solution that processes all provided CAPTCHAs, correctly reports accuracy, and compares filename vs OCR result. Code is clear, well-commented, and replicable.
• 1 point: Functional OCR implementation but with minor issues (e.g., some images skipped, no accuracy % computed).
• 0 points: OCR implementation missing or does not work.

---

Proposed CAPTCHA Redesign and Justification (1.5 points)

• 1.5 points: Proposes a thoughtful CAPTCHA improvement that balances security (OCR resistance) and usability. Includes clear explanation of design choices, why they impact OCR, and usability trade-offs.
• 1 point: Proposes a valid improvement but lacks clear reasoning or ignores usability vs security balance.
• 0 points: No improvement proposed, or proposed design isn’t feasible.

---

Evaluation of New CAPTCHA Design (1.5 points)

• 1.5 points: Tests OCR on new CAPTCHA, compares OCR success rate between old and new designs, and provides insightful reflection on whether security increased and how usability was affected.
• 1 point: Provides some comparison but lacks analysis of usable security or impact on humans.
• 0 points: No evaluation or comparison.

---

Clarity and Presentation (1 point)

• 1 point: Report is clear, well-structured, follows instructions, and free of major writing errors.
• 0.5 points: Report is understandable but contains some clarity/formatting issues.
• 0 points: Report unclear or poorly written.

---

Total: 6 Points

• 5–6 points: Excellent work; demonstrates full understanding of both technical and usability aspects of CAPTCHA design, breaking, and evaluation.
• 3–4 points: Good work; minor omissions in analysis or explanation.
• 1–2 points: Incomplete or lacking depth in solving, redesign, or evaluation.
• 0 points: Major components missing or incorrect.