Doesn't read two columns PDF correctly, thinks it is a one column

[hhblaze]

2302.14502.pdf

                using (UglyToad.PdfPig.PdfDocument document = UglyToad.PdfPig.PdfDocument.Open(pdfFilePath))
                {
                    foreach (Page page in document.GetPages())
                    {                
                        foreach (Word word in page.GetWords())
                        {
                            Console.Write(word.Text + " ");
                        }
                    }
                }

[BobLd]

This page has a complex layout, please have a look at the Page Segmenter section in the wiki.

You can try the below, should solve your problem:

using (var document = PdfDocument.Open("document.pdf"))
{
   for (var i = 0; i < document.NumberOfPages; i++)
   {
	   var page = document.GetPage(i + 1);
	   var words = page.GetWords();

	   // Use default parameters
	   // - mode of letters' height and width used as gap size
	   // - no minimum block width 
	   var blocks = RecursiveXYCut.Instance.GetBlocks(words);

	   foreach (var block in blocks)
	   {
		   foreach (TextLine line in block.TextLines)
		   {
			   foreach (Word word in line.Words)
			   {
				   Console.Write(word.Text + " ");
			   }
		   }
	   }
   }
}

If the blocks are not in the correct order, have a look at the Reading Order Detectors section.

[hhblaze]

Thank you, first. I have an automatic system, and have no ability to take care about each doc layout explicitly. That's the main problem. Old iTextSharp correctly solves this document, iText7 acts the same like PdfPig, what a "nice"... PyPdf works also correctly, but not C#. So, saying shortly, PdfPig must automatically adjust the layout.

[hhblaze]

But this reading by block - works, so I can assume that it will work for any pdf? As I understand, there is a problem to identify the block in pdf - I have no experience in PDF protocol - but you are trying to use heuristic to determine it. But OK, will give a try, thank you!

[BobLd]

@hhblaze yes pdf does not have a notion of 'paragraph' or 'line' of text... every pdf library will have some heuristic to reconstruct the actual text. You could also have a look at how the old iText does it and implement it with PdfPig

[hhblaze]

This one also works good for different docs I have for tests:

 static void testAdv()
 {
     var sb = new StringBuilder();
     string pdfFilePath = @"...\2302.14502.pdf";

     using (var document = UglyToad.PdfPig.PdfDocument.Open(pdfFilePath))
     {
         foreach (var page in document.GetPages())
         {
             // 0. Preprocessing
             var letters = page.Letters; // no preprocessing

             // 1. Extract words
             //var wordExtractor = NearestNeighbourWordExtractor.Instance;
             
             var wordExtractorOptions = new NearestNeighbourWordExtractor.NearestNeighbourWordExtractorOptions()
             {
                 Filter = (pivot, candidate) =>
                 {
                     // check if white space (default implementation of 'Filter')
                     if (string.IsNullOrWhiteSpace(candidate.Value))
                     {
                         // pivot and candidate letters cannot belong to the same word 
                         // if candidate letter is null or white space.
                         // ('FilterPivot' already checks if the pivot is null or white space by default)
                         return false;
                     }

                     // check for height difference
                     var maxHeight = Math.Max(pivot.PointSize, candidate.PointSize);
                     var minHeight = Math.Min(pivot.PointSize, candidate.PointSize);
                     if (minHeight != 0 && maxHeight / minHeight > 2.0)
                     {
                         // pivot and candidate letters cannot belong to the same word 
                         // if one letter is more than twice the size of the other.
                         return false;
                     }

                     // check for colour difference
                     var pivotRgb = pivot.Color.ToRGBValues();
                     var candidateRgb = candidate.Color.ToRGBValues();
                     if (!pivotRgb.Equals(candidateRgb))
                     {
                         // pivot and candidate letters cannot belong to the same word 
                         // if they don't have the same colour.
                         return false;
                     }

                     return true;
                 }
             };

             var wordExtractor = new NearestNeighbourWordExtractor(wordExtractorOptions);

             var words = wordExtractor.GetWords(letters);

             // 2. Segment page
             //var pageSegmenter = DocstrumBoundingBoxes.Instance;
             var pageSegmenterOptions = new DocstrumBoundingBoxes.DocstrumBoundingBoxesOptions()
             {

             };

             var pageSegmenter = new DocstrumBoundingBoxes(pageSegmenterOptions);

             var textBlocks = pageSegmenter.GetBlocks(words);

             // 3. Postprocessing
             var readingOrder = UnsupervisedReadingOrderDetector.Instance;
             var orderedTextBlocks = readingOrder.Get(textBlocks);

             // 4. Extract text
             foreach (var block in orderedTextBlocks)
             {
                 sb.Append(block.Text.Normalize(NormalizationForm.FormKC)); // normalise text
                 sb.AppendLine();
             }

             sb.AppendLine();
         }
     }

     Console.WriteLine(sb.ToString());
 }