java - What is the best way to check each link of a website? -

-

I want to create a crawler that follows every link on a site, and the URL checks to see if it Works or not. Now my code is url.openStream () .

, what is the best way to create a crawler?

Use an HTML parser. 

 set  & lt; String & gt; Valid links = new hashset & lt; String & gt; (); Set up & lt; String & gt; InvalidLinks = NewHashet & lt; String & gt; (); Document Document = Jsoup.connect ("http://example.com") .get (); Element link = document. Select ("a"); (String link: links) {string url = link.bsr ("href"); If (! Legitimate link link (url) & illegal link link (url)) {iit status code = jsoup.connect (url) .execute (). Statuscode (); If (200 & lt; = statusCode & amp; status code & lt; 400) {validLinks.add (url); } And {invalidLinks.add (url); }} Hold (exception e) {invalidLinks.add (url); }}}   
 You want to send a HEAD instead of that loop to make it more efficient, but then you instead use  URLConnection  as Jason Do not support it by design (a HEAD does not give any content)

Comments

Post a Comment

Popular posts from this blog

mysql - BLOB/TEXT column 'value' used in key specification without a key length -

-
I have developed an extension which works up to 1.6 on Magren (I'm trying Enterprise Edition, And I think the community is the same problem, because it is the same code). In my install script, I see the $ installer-> gt; CreateEntityTables ($ this- & gt; getTable ('alphanum / info')); The installation is done until it is not in the _text unit table. It crashed there! It turns out that when I log in to SQL and run it via PHPmyadmin, then this error is: Blob / Text column 'value' is used without the key 'key' . I saw the code there, and this is what is trying to create an index on the value column: -> addIndex ($ this- & gt; getIdxName ($ eavTableName, array ( 'attribute_id array (' attribute_id ',' value ')) - & gt; addIndex ($ this- & gt; getIdxName ($ eavTableName, array (' entity_type_id ' , 'Value')), array ('entity_type_id', 'value')) If there is no if statement is n...
Read more

c# - Using Vici cool Storage with monodroid -

-
Oh, for the moment I am not sure how it is officially supported, but for the success of the people Using Monodroid Vic Cool Storage I am able to leave and compile assemblies in my project, but while trying to use some classes, the errors of time have to be gathered. Especially when trying to connect like the example for Mono Touch string dbName = path. Cobain (environment. Gatefolder path (environment specialfolder personal), "mydb.db3"); // The following line will be called coolestation, where is the database, // to make it if it does not exist, and call a delegate that creates the necessary tables (only when the database file was newly created //) CSConfig.SetDB (DbName, true, () => {CSDatabase.ExecuteNonQuery ("Subscriber Integer Primary Key Attribute, Name Text (50) No, Number, Date First Text (30)");); There is no difference when I try to use the methods of CSConfig and when I cscfif If I try to pass 3 aggregates to STDB (3) then I get the invalid ...
Read more

c# - Confused over DLL entry points (entry point not found exception) -

-
I'm learning how to use DLL in C #. I have a very simple DLL to do my basic test. // using the MainForm.cs system; Using System.Collections.Generic; Using System.ComponentModel; Using System.Data; Using System.Drawing; Using System.Linq; Using System.Text; Using System.Windows.Forms; Using System.Runtime.InteropServices; Namespace DLL_Test {public partial class} Form 1: Form {{DllImport ("TestDLL.dll", EntryPoint = "? @ @ YGHHH Add @, Accurate Spelling = True, Calling Communication = Calling Conference. STCL}] Add public static extern int ( Int a, int b); public form 1 () {initialization ();} private zeros button 1_Click (object sender, eventArgs e) {int num; try {num = add (2, 3); RichTextBox1.AppendText (num. ToString () + "\ n");} Hold (DllNotFoundException Pre) {Message Box.Show (ex.ToString ());} Hold (EntryPointNotFoundException Pre) {Message Box.Show (ex.ToString ())}} }} more DLL code: // TestDLL.cpp __declspec (dllexport) Add int __std...
Read more