Behind the scenes

About Log4j and Log4shell

Christian Grobmeier
mastodon.social/@grobmeier
https://grobmeier.solutions

Your Image Description

Who are you?

Your Image Description

At night...

Authoring a book on Java Logging for Manning
Apache Software Foundation Member
Current VP Data Privacy
Ex-VP Logging Services
Committer to log4php, log4j 1, minor contributions to log4j2
Contributions to many other Open Source projects

About the ASF

Many citizens in the ASF

Num3ers

ASF Numbers Overview
Language Distribution
ASF Statistics
Source: 2024 projects.apache.org

How does the ASF work?

  • Meritocracy / Docracy
  • Building consensous
  • Voting
  • Mailing lists

Open Source should be fun

Funding for ~13 years of service:

0 USD

  • Pre-Incident: a few bucks
  • After Incident: some donations via GitHub or directly
  • One year after incident: Some GitHub sponsoring
  • Afterwards: STF Funding

Let's talk about Log4(s)hell

Log4j versions

2001 - 2015: Log4j 1 (unaffected)
➡ 2014 - 202x: Log4j 2 (affected)

Log4j Team

6 active maintainers
Roughly 20 inactive maintainers
Low profile until 2021

Log4shell

The feature was created in July 2013, before the first release of Log4j 2.
It was feature 313 from an outside contributor.

Timeline

Why didn't you see this?

Want to fix your security knowledge?

Detailed timeline

  • 2021-11-24 - First report of issue ❓
  • 2021-11-25 - Issue acknowledged (affected: Struts, Flink...) 😱
  • 2021-11-29 - Issue understood and solution discussed 👍
  • 2021-12-05 - Issue solved - Release vote started 👍
  • 2021-12-08 - (Public) security groups started to talk about it on social media 😱
  • 2021-12-09 - Reporter was able to bypass the fix, vote cancelled 😱
  • 2021-12-09 - New RC prepared, more vulnerabilites 😱
  • 2021-12-10 - Log4j 2.15.0 was released (fast track) 😱
  • 2021-12-09 - Even more vulnerabilites 😱😱😱

What people thought we would do:

Detailed timeline

  • Tons of private messages
  • Meetings on Slack
  • Releasing multiple versions
  • Dealing with many other security reports
  • 😱😱😱Some of us worked almost 24h in a row. 😱😱😱

Constructive Feedback

People don’t know if they have a problem.
Spring Boot by default comes with 115 dependencies.

Want to fix your security knowledge?

Maven Central Statistics (Sonatype) from 20.03.2023. Down to 18% as of 2024.

FAQ around 2021

  • Should I use Log4j 1?
  • Why did you not restart Log4j 1?
  • Can you fix 10 year old “security issues” too?
  • I am playing Minecraft, is this a problem?
  • Press: what is going to explode?

Other lessons learned

When you report security issues, your government may harm you.
Chinese gov punished Alibaba for reporting the issue. Germany party CDU sued the reporter of the CDU Connect problem.
Don't rely on developers to update their dependencies.

My personal key questions

  • Who is developing the software?
  • Is it a single-person project?
  • Does the developer regularly contribute?
  • Can I fix it myself?
  • Is somebody paying the developer?

Sustainable Open Source?

Money may be a problem

  • Paid versus unpaid
  • Speed
  • People are people
  • Taxes

Today

  • Log4j 2 is "safe"
  • Better quality due to funding
  • Improved upgrade paths
  • New committers
  • security@logging.apache.org

Thank you

Christian Grobmeier
cg@grobmeier.de
https://mastodon.social/@grobmeier
https://bsky.app/profile/grobmeier.bsky.social
https://www.linkedin.com/in/grobmeier/
https://grobmeier.solutions

Credits