When attempting to read or process a PDF protected by AES-256 encryption in iText5/iTextSharp, or even older versions of iText, you will likely see this error. This is probably due to the document being PDF 2.0 compliant, since this encryption type was not supported in earlier versions of the PDF specification. You will need to use a version of iText that supports PDF 2.0 to be able to recognize this type of encryption. PDF 2.0 support was first introduced in iText Core 7.1, though newer versions have support for the later PDF 2.0 ISO extension standards.
All versions of iText since the release of iText 5 in 2009 have been released under the same dual licensing scheme: AGPL and commercial. So, if you need to upgrade we recommend using the most recent version of iText Core to support even newer encryption types, PDF 2.0 extension standards, and more.
You should also be aware that if you have never encountered this error when dealing with encrypted PDFs, then these documents are probably not secure by modern standards. For more information on why this is, and exactly what causes the error message, refer to this article: How to solve “Unknown encryption type R = 6” errors.