{"id":2913,"date":"2017-08-02T16:08:19","date_gmt":"2017-08-02T15:08:19","guid":{"rendered":"https:\/\/dn-www.azurewebsites.net\/2017\/08\/02\/reposcanner\/"},"modified":"2024-02-06T10:17:22","modified_gmt":"2024-02-06T10:17:22","slug":"reposcanner","status":"publish","type":"post","link":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/","title":{"rendered":"Reposcanner"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"2913\" class=\"elementor elementor-2913\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-121c0b00 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"121c0b00\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-e40b9c6\" data-id=\"e40b9c6\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-4fe24f05 elementor-widget elementor-widget-text-editor\" data-id=\"4fe24f05\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><a href=\"https:\/\/www.github.com\/Dionach\/reposcanner\">Reposcanner<\/a> is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by <a href=\"https:\/\/github.com\/dxa4481\/truffleHog\">truffleHog<\/a>. Sensitive information like this often gets included in the earlier stages of the development process (or accidentally), and is generally removed before the application or source code is released. However, since Git keeps a history of all changes, by going back through these commits, we can scan back through the commit history to obtain information that has been removed in the latest version. The basic flow of reposcanner is as follows:<\/p>\n<ul>\n<li>Try and clone the repository if a remote URL is given<\/li>\n<li>Get the active branch (or all branches if the -a option is given)<\/li>\n<li>Create a diff for each commit in the select branch(es)<\/li>\n<li>Ignore any known boring string patterns and files names\/extensions<\/li>\n<li>Extract any long hexadecimal or base64 strings<\/li>\n<li>Calculate the entropy of these strings<\/li>\n<li>If the entropy is high enough, and it&#8217;s not been seen before, store the string<\/li>\n<li>Output all strings that are found<\/li>\n<\/ul>\n<p>&nbsp;<br>The hardest step is trying to identify &#8220;interesting&#8221; strings, without ending up with too many false positives. Reposcanner has some known patterns of boring strings which it ignores, and you can tweak the minimum entropy to report if you&#8217;re getting too many false positive (the current value was obtained through some trial and error). A possible future option might be to also search for interesting strings (such as &#8220;api_key = foo&#8221;, or connection strings). Unlike truffleHog, which shows the entire diffs to give context, reposcanner has a much more concise output, which only shows you the relevant line, along with the commit information so that you can go and examine the commit yourself if the string looks interesting. This makes the output much more manageable, especially when scanning larger repositories. Scanning some randomly selected repositories in GitHub resulted in the expected interesting strings, including:<\/p>\n<ul>\n<li>API keys third party services, in an employee financial bonus scheme<\/li>\n<li>Application and database passwords<\/li>\n<li>A SQL database backup<\/li>\n<\/ul>\n<p>&nbsp;<br>This is a serious risk for companies when internally developed projects are released to the public &#8211; developers are less likely to be careful with their commits to an internal project compared to a publicly available one, and this increases the likelihood of inappropriate files making their way into the version control system. It can also reflect badly on a company if you have unprofessional code, comments or commit messages &#8211; a message like &#8220;accidentally deleted database&#8221; doesn&#8217;t tend to inspire confidence.<\/p>\n<p>Going through the commit history and trying to sanitise is likely to be unfeasible, unless it&#8217;s a trivially sized repository, so the approach that most organisations take is just to completely wipe the commit history &#8211; either by creating a fresh repo and copying the files into it, or destroying the entire history with a rebase. While this provides a degree of protection from inappropriate commit messages or data being leaked, it does also destroy the development history of the repository, which is very valuable to developers when trying to fix bugs, or to understand why certain decisions have been made in the development process. As always, it&#8217;s the trade-off between security and convenience.<\/p>\n<p>Besides destroying the repo history, the best thing that you can do to protect against these issues to have secure development practices from the start, even for projects that you&#8217;re never anticipating releasing. This should include making sure that sensitive information is never committed into source control, and of course, trying to keep comments and commit messages (reasonably) professional.<\/p>\n<p>The Reposcanner code is available on the Dionach GitHub at <a href=\"https:\/\/github.com\/Dionach\/reposcanner\">https:\/\/github.com\/Dionach\/reposcanner<\/a> &#8211; pull request are welcome as always.<\/p><p><br><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by truffleHog. Sensitive information like this often gets included in the earlier stages of the development process (or accidentally), and is generally removed before the application or source code is released. However, since [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"content-type":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[214,207],"class_list":["post-2913","post","type-post","status-publish","format-standard","hentry","category-researchblog","tag-tools","tag-web_applications","wpbf-post"],"contentshake_article_id":"","yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Reposcanner<\/title>\n<meta name=\"description\" content=\"Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Reposcanner\" \/>\n<meta property=\"og:description\" content=\"Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/\" \/>\n<meta property=\"og:site_name\" content=\"Dionach\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/dionachcyber\" \/>\n<meta property=\"article:published_time\" content=\"2017-08-02T15:08:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-06T10:17:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/dionach.development-visionsharp.co.uk\/wp-content\/uploads\/2025\/02\/cropped-Dionach-vertical-col-yel-nomios-black-1.jpg?fit=512%2C512&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Dionach Admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@dionachcyber\" \/>\n<meta name=\"twitter:site\" content=\"@dionachcyber\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dionach Admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/\"},\"author\":{\"name\":\"Dionach Admin\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#\\\/schema\\\/person\\\/effca060e22bfa3cc6cd03f74a50fdb4\"},\"headline\":\"Reposcanner\",\"datePublished\":\"2017-08-02T15:08:19+00:00\",\"dateModified\":\"2024-02-06T10:17:22+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/\"},\"wordCount\":623,\"publisher\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#organization\"},\"keywords\":[\"tools\",\"web applications\"],\"articleSection\":[\"researchblog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/\",\"url\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/\",\"name\":\"Reposcanner\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#website\"},\"datePublished\":\"2017-08-02T15:08:19+00:00\",\"dateModified\":\"2024-02-06T10:17:22+00:00\",\"description\":\"Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/reposcanner\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/homepage-usa\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Reposcanner\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#website\",\"url\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/\",\"name\":\"Dionach\",\"description\":\"Real Security in a Virtual World\",\"publisher\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#organization\",\"name\":\"Dionach\",\"url\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/dionach.com\\\/wp-content\\\/uploads\\\/2025\\\/02\\\/cropped-Dionach-vertical-col-yel-nomios-black-1.jpg\",\"contentUrl\":\"https:\\\/\\\/dionach.com\\\/wp-content\\\/uploads\\\/2025\\\/02\\\/cropped-Dionach-vertical-col-yel-nomios-black-1.jpg\",\"width\":512,\"height\":512,\"caption\":\"Dionach\"},\"image\":{\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/dionachcyber\",\"https:\\\/\\\/x.com\\\/dionachcyber\",\"https:\\\/\\\/uk.linkedin.com\\\/company\\\/dionach-ltd\",\"https:\\\/\\\/www.instagram.com\\\/dionachcyber\\\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/dionach.development-visionsharp.co.uk\\\/en-us\\\/#\\\/schema\\\/person\\\/effca060e22bfa3cc6cd03f74a50fdb4\",\"name\":\"Dionach Admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3061726a64a760303f6ea8f0976d3e8e0a6997b4da543be9a650b81584b4e79e?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3061726a64a760303f6ea8f0976d3e8e0a6997b4da543be9a650b81584b4e79e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/3061726a64a760303f6ea8f0976d3e8e0a6997b4da543be9a650b81584b4e79e?s=96&d=mm&r=g\",\"caption\":\"Dionach Admin\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Reposcanner","description":"Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Reposcanner","og_description":"Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by","og_url":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/","og_site_name":"Dionach","article_publisher":"https:\/\/www.facebook.com\/dionachcyber","article_published_time":"2017-08-02T15:08:19+00:00","article_modified_time":"2024-02-06T10:17:22+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/i0.wp.com\/dionach.development-visionsharp.co.uk\/wp-content\/uploads\/2025\/02\/cropped-Dionach-vertical-col-yel-nomios-black-1.jpg?fit=512%2C512&ssl=1","type":"image\/jpeg"}],"author":"Dionach Admin","twitter_card":"summary_large_image","twitter_creator":"@dionachcyber","twitter_site":"@dionachcyber","twitter_misc":{"Written by":"Dionach Admin","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/#article","isPartOf":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/"},"author":{"name":"Dionach Admin","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#\/schema\/person\/effca060e22bfa3cc6cd03f74a50fdb4"},"headline":"Reposcanner","datePublished":"2017-08-02T15:08:19+00:00","dateModified":"2024-02-06T10:17:22+00:00","mainEntityOfPage":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/"},"wordCount":623,"publisher":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#organization"},"keywords":["tools","web applications"],"articleSection":["researchblog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/","url":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/","name":"Reposcanner","isPartOf":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#website"},"datePublished":"2017-08-02T15:08:19+00:00","dateModified":"2024-02-06T10:17:22+00:00","description":"Reposcanner is a Python script designed to scan Git repositories looking for interesting strings, such as API keys or hard-coded passwords, inspired by","breadcrumb":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/reposcanner\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/homepage-usa\/"},{"@type":"ListItem","position":2,"name":"Reposcanner"}]},{"@type":"WebSite","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#website","url":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/","name":"Dionach","description":"Real Security in a Virtual World","publisher":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#organization","name":"Dionach","url":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#\/schema\/logo\/image\/","url":"https:\/\/dionach.com\/wp-content\/uploads\/2025\/02\/cropped-Dionach-vertical-col-yel-nomios-black-1.jpg","contentUrl":"https:\/\/dionach.com\/wp-content\/uploads\/2025\/02\/cropped-Dionach-vertical-col-yel-nomios-black-1.jpg","width":512,"height":512,"caption":"Dionach"},"image":{"@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/dionachcyber","https:\/\/x.com\/dionachcyber","https:\/\/uk.linkedin.com\/company\/dionach-ltd","https:\/\/www.instagram.com\/dionachcyber\/"]},{"@type":"Person","@id":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/#\/schema\/person\/effca060e22bfa3cc6cd03f74a50fdb4","name":"Dionach Admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/3061726a64a760303f6ea8f0976d3e8e0a6997b4da543be9a650b81584b4e79e?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/3061726a64a760303f6ea8f0976d3e8e0a6997b4da543be9a650b81584b4e79e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3061726a64a760303f6ea8f0976d3e8e0a6997b4da543be9a650b81584b4e79e?s=96&d=mm&r=g","caption":"Dionach Admin"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/ph4Ojq-KZ","_links":{"self":[{"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/posts\/2913","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/comments?post=2913"}],"version-history":[{"count":0,"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/posts\/2913\/revisions"}],"wp:attachment":[{"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/media?parent=2913"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/categories?post=2913"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dionach.development-visionsharp.co.uk\/en-us\/wp-json\/wp\/v2\/tags?post=2913"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}