Scan for and Detect Personally Identifiable Information

This skill builds comprehensive PII detection systems that identify sensitive data across formats and support GDPR, CCPA, and HIPAA compliance. Detection layers five methods: pattern-based regex for structured data (SSNs, credit cards, phone numbers), context-aware scanning of surrounding text and field names, statistical entropy analysis for tokens that might be encrypted or hashed PII, ML-based classification via named entity recognition, and format validation via checksum algorithms. Every detection carries a confidence score from 0.0 to 1.0, combining multiple methods for higher accuracy, reported against configurable sensitivity thresholds, with false-positive rates tracked to tune those thresholds over time.

Use it when scanning datasets, logs, or pipelines for regulated personal data that needs both pattern accuracy and semantic context, not a single regex sweep with no validation. It is not meant to store what it finds: the compliance guidance requires data minimization - avoiding storing actual PII values in audit logs - and configurable sensitivity thresholds per regulatory requirement rather than one fixed setting for every use case.

Given a data record, the pattern detector matches four named PII types with format validators - a Social Security number pattern excluding known-invalid ranges, a credit card pattern validated by the Luhn checksum algorithm, an email pattern, and a phone pattern - each starting from a base confidence that's halved on a failed validator and boosted by up to 0.2 from contextual scoring. Contextual scoring weights field-name keyword matches, surrounding-text phrases (like "personal information" or "billing address"), and pattern matches. An ML layer adds spaCy named-entity recognition (person, organization, location, date, and money entities) alongside a transformer-based NER pipeline. The comprehensive scanner runs all three layers per field, filters by a configurable confidence threshold, and computes an overall risk score as the fraction of scanned fields that triggered a detection.

### Example integration with data pipeline
class PIIAwarePipeline:
    def __init__(self):
        self.scanner = ComprehensivePIIScanner()
        self.quarantine_storage = PIIQuarantineStorage()
    
    def process_batch(self, records: List[Dict]) -> Dict:
        clean_records = []
        flagged_records = []
        
        for record in records:
            scan_result = self.scanner.scan_data(record)
            
            if scan_result['risk_score'] > 0.3:  # High PII risk
                self.quarantine_storage.store(record, scan_result)
                flagged_records.append(record['id'])
            else:
                clean_records.append(record)
        
        return {
            'processed': len(clean_records),
            'flagged': len(flagged_records),
            'flagged_ids': flagged_records
        }

Batch pipeline integration scans each record, quarantines any record whose risk score exceeds a configured threshold, and reports processed versus flagged counts with flagged record IDs. Performance practices layer on top: compiling regex patterns once, early termination once the confidence threshold is met, batching records for ML inference, and caching validation results for repeated values. False-positive reduction relies on validation algorithms, context analysis, a whitelist of known non-PII patterns, multi-method confirmation before reporting high confidence, and human-in-the-loop review for edge cases.

Data engineering and privacy teams building PII detection into data pipelines who need layered accuracy, not a single regex pass - with audit trails, configurable per-regulation sensitivity, regularly updated detection patterns, and tracked precision, recall, and processing-time metrics.