얼거스 개발 일지

에러 코드

C:\Users\xh\.jdks\openjdk-19.0.2\bin\java.exe -XX:TieredStopAtLevel=1 -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true "-Dmanagement.endpoints.jmx.exposure.include=*" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2022.3.1\lib\idea_rt.jar=49352:C:\Program Files\JetBrains\IntelliJ IDEA 2022.3.1\bin" -Dfile.encoding=UTF-8 -Dsun.stdout.encoding=UTF-8 -Dsun.stderr.encoding=UTF-8 -classpath C:\Users\xh\Documents\zb-dividend\financial\out\production\classes;C:\Users\xh\Documents\zb-dividend\financial\out\production\resources;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.projectlombok\lombok\1.18.22\9c08ea24c6eb714e2d6170e8122c069a0ba9aacf\lombok-1.18.22.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-web\2.5.6\46b479490170914f7477b96a21241183b181c24d\spring-boot-starter-web-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-data-jpa\2.5.6\8d7fe99c33e09390316749614d9795d80b49207b\spring-boot-starter-data-jpa-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-security\2.5.6\af5827b9e08ea631fa213cccd1144fbdfee32896\spring-boot-starter-security-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.jsoup\jsoup\1.7.2\d7e275ba05aa380ca254f72d0c0ffebaedc3adcf\jsoup-1.7.2.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-json\2.5.6\6ef5a7087e18ed4f3736c8752440ecd489c36a4d\spring-boot-starter-json-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter\2.5.6\d5d1fada1afe9a808abf48da7066a993cf679aa\spring-boot-starter-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-tomcat\2.5.6\6d1a04a727d9d09b99207864ceb0a4567e53730a\spring-boot-starter-tomcat-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-webmvc\5.3.12\3d92ad6c28bfa5923183f328f5bfa1e39ec32714\spring-webmvc-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-web\5.3.12\78991a50d17da49bddc4987a2cc8b83d46c402a7\spring-web-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-aop\2.5.6\c5db1260ecf447f55419f1a17da75a42f211aca3\spring-boot-starter-aop-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-jdbc\2.5.6\cf01e787378c2d30b695f0c9f76fb48a6b490984\spring-boot-starter-jdbc-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\jakarta.transaction\jakarta.transaction-api\1.3.3\c4179d48720a1e87202115fbed6089bdc4195405\jakarta.transaction-api-1.3.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\jakarta.persistence\jakarta.persistence-api\2.2.3\8f6ea5daedc614f07a3654a455660145286f024e\jakarta.persistence-api-2.2.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.hibernate\hibernate-core\5.4.32.Final\99a5e10bf455337014c190e141ec631e9ff71663\hibernate-core-5.4.32.Final.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.data\spring-data-jpa\2.5.6\8e0ec2f54f3fcda49dfb3123f3a40f34b55df92a\spring-data-jpa-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-aspects\5.3.12\3cccc3052c6973c059eb2be7c4baf0b9558d49b7\spring-aspects-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.security\spring-security-web\5.5.3\2d2b773e2af5b5984852db8857a77175ce4e1104\spring-security-web-5.5.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.security\spring-security-config\5.5.3\106b6a1af7460d64fab64ba5bbfe3f52f0eec139\spring-security-config-5.5.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-aop\5.3.12\882db41939109e96f4c78cd5c0931cc4aebc3d58\spring-aop-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.datatype\jackson-datatype-jsr310\2.12.5\a0a9870b681a72789c5c6bdc380e45ab719c6aa3\jackson-datatype-jsr310-2.12.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.module\jackson-module-parameter-names\2.12.5\2c85c2036d0851425a260c01eb5f7ddbed1eeb00\jackson-module-parameter-names-2.12.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.datatype\jackson-datatype-jdk8\2.12.5\6b2f79547d217ad50dfc5b57af7444a3aa583b43\jackson-datatype-jdk8-2.12.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-databind\2.12.5\b064cf057f23d3d35390328c5030847efeffedde\jackson-databind-2.12.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-autoconfigure\2.5.6\b9f4016180c5242530da465561ff25c7cac14bf3\spring-boot-autoconfigure-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot\2.5.6\d8c6b97fd3182fb6d7d06ebf710cd9ccabc83b89\spring-boot-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter-logging\2.5.6\a900356a11b1a41f4277136f1d13ce7a13f43b3c\spring-boot-starter-logging-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\jakarta.annotation\jakarta.annotation-api\1.3.5\59eb84ee0d616332ff44aba065f3888cf002cd2d\jakarta.annotation-api-1.3.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-core\5.3.12\662e6536968246af9baa84fbac2d3eb56a04fda9\spring-core-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.yaml\snakeyaml\1.28\7cae037c3014350c923776548e71c9feb7a69259\snakeyaml-1.28.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.apache.tomcat.embed\tomcat-embed-websocket\9.0.54\ae018906cecb818a8c6f2316d7b0793beadf6609\tomcat-embed-websocket-9.0.54.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.apache.tomcat.embed\tomcat-embed-core\9.0.54\34322c731b2394ea13681cfae0be9cd72f46f88d\tomcat-embed-core-9.0.54.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.apache.tomcat.embed\tomcat-embed-el\9.0.54\9edb062d38d0fd8a165289f44b28b3b0e0e11ed7\tomcat-embed-el-9.0.54.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-context\5.3.12\d5f5f044e05109b7f3337ea2cf692fd62d1ecbb6\spring-context-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-beans\5.3.12\caaa1d489bce88d6aa01ddd255ad5046acf8f282\spring-beans-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-expression\5.3.12\50c82e995b3b8e20a3f313b4356237db5a26e14a\spring-expression-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.aspectj\aspectjweaver\1.9.7\158f5c255cd3e4408e795b79f7c3fbae9b53b7ca\aspectjweaver-1.9.7.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-jdbc\5.3.12\957d6ddc80fbf52d965e6af90ddd0dccfed42d7d\spring-jdbc-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.zaxxer\HikariCP\4.0.3\107cbdf0db6780a065f895ae9d8fbf3bb0e1c21f\HikariCP-4.0.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.hibernate.common\hibernate-commons-annotations\5.1.2.Final\e59ffdbc6ad09eeb33507b39ffcf287679a498c8\hibernate-commons-annotations-5.1.2.Final.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.jboss.logging\jboss-logging\3.4.2.Final\e517b8a93dd9962ed5481345e4d262fdd47c4217\jboss-logging-3.4.2.Final.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.javassist\javassist\3.27.0-GA\f63e6aa899e15eca8fdaa402a79af4c417252213\javassist-3.27.0-GA.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\net.bytebuddy\byte-buddy\1.10.22\ef45d7e2cd1c600d279704f492ed5ce2ceb6cdb5\byte-buddy-1.10.22.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\antlr\antlr\2.7.7\83cd2cd674a217ade95a4bb83a8a14f351f48bd0\antlr-2.7.7.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.jboss\jandex\2.2.3.Final\d3865101f0666b63586683bd811d754517f331ab\jandex-2.2.3.Final.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml\classmate\1.5.1\3fe0bed568c62df5e89f4f174c101eab25345b6c\classmate-1.5.1.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.dom4j\dom4j\2.1.3\a75914155a9f5808963170ec20653668a2ffd2fd\dom4j-2.1.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.glassfish.jaxb\jaxb-runtime\2.3.5\a169a961a2bb9ac69517ec1005e451becf5cdfab\jaxb-runtime-2.3.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-orm\5.3.12\2881f9e71889b35fa3785bf67706a201cea93004\spring-orm-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.data\spring-data-commons\2.5.6\15a2384f4eaf7fee512fb295174f6c0fb6c55ee1\spring-data-commons-2.5.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-tx\5.3.12\7f2e61a22682baa22ed5bef0724a4386c41477cb\spring-tx-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.slf4j\slf4j-api\1.7.32\cdcff33940d9f2de763bc41ea05a0be5941176c3\slf4j-api-1.7.32.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.security\spring-security-core\5.5.3\82152ffbb7d248e0903732c74e1578317d8dc8de\spring-security-core-5.5.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-annotations\2.12.5\52d929d5bb21d0186fe24c09624cc3ee4bafc3b3\jackson-annotations-2.12.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.12.5\725e364cc71b80e60fa450bd06d75cdea7fb2d59\jackson-core-2.12.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\ch.qos.logback\logback-classic\1.2.6\b09efa852337fa0dd9859614389eec58dc287116\logback-classic-1.2.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-to-slf4j\2.14.1\ce8a86a3f50a4304749828ce68e7478cafbc8039\log4j-to-slf4j-2.14.1.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.slf4j\jul-to-slf4j\1.7.32\8a055c04ab44e8e8326901cadf89080721348bdb\jul-to-slf4j-1.7.32.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework\spring-jcl\5.3.12\2b5f5bb4a78af879bd174ceff5226da3f014ab9d\spring-jcl-5.3.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\jakarta.xml.bind\jakarta.xml.bind-api\2.3.3\48e3b9cfc10752fba3521d6511f4165bea951801\jakarta.xml.bind-api-2.3.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.glassfish.jaxb\txw2\2.3.5\ec8930fa62e7b1758b1664d135f50c7abe86a4a3\txw2-2.3.5.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.sun.istack\istack-commons-runtime\3.0.12\cbbe1a62b0cc6c85972e99d52aaee350153dc530\istack-commons-runtime-3.0.12.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.springframework.security\spring-security-crypto\5.5.3\45fc09a7a2484ef843a9db4652e6ff984bc2e537\spring-security-crypto-5.5.3.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\ch.qos.logback\logback-core\1.2.6\25be1abb32e870ff042e698a799b56587e0dca9a\logback-core-1.2.6.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\org.apache.logging.log4j\log4j-api\2.14.1\cd8858fbbde69f46bce8db1152c18a43328aae78\log4j-api-2.14.1.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.h2database\h2\1.4.200\f7533fe7cb8e99c87a43d325a77b4b678ad9031a\h2-1.4.200.jar;C:\Users\xh\.gradle\caches\modules-2\files-2.1\com.sun.activation\jakarta.activation\1.2.2\74548703f9851017ce2f556066659438019e7eb5\jakarta.activation-1.2.2.jar kim.zhyun.financial.FinancialApplication
Exception in thread "main" java.lang.RuntimeException: org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=https://www.google.com/search?q=%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91+%EC%98%81%EC%96%B4%EB%A1%9C&oq=%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91+%EC%98%81%EC%96%B4%EB%A1%9C&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCDMyMzBqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8
	at kim.zhyun.financial.FinancialApplication.main(FinancialApplication.java:51)
Caused by: org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=https://www.google.com/search?q=%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91+%EC%98%81%EC%96%B4%EB%A1%9C&oq=%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91+%EC%98%81%EC%96%B4%EB%A1%9C&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCDMyMzBqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8
	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:435)
	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:410)
	at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:164)
	at org.jsoup.helper.HttpConnection.get(HttpConnection.java:153)
	at kim.zhyun.financial.FinancialApplication.main(FinancialApplication.java:25)

종료 코드 1(으)로 완료된 프로세스

스크래핑을 연습해보려고 시도한 순간 에러를 만났다! ㅋㅋ

강의 따라할 때는 에러가 안났는데

복습해보려고 첫 시도한 순간 에러가 터졌다.

정말 나란 사람 .. ㅋㅋㅋㅋㅋ

핵심 에러 코드는 아래와 같다.

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403,

도움 받은 곳 : javacodeexamples

도움 받은 사이트에 의하면

HTTP 403 상태 코드는 클라이언트의 웹 페이지나 리소스 요청에 대한 응답으로 웹 서버가 서버에 도달하고 요청을 이해했지만, 더 이상의 조치를 취하지 않겠다는 것을 나타내기 위해 반환하는 것입니다.

Jsoup을 사용하면서 403 오류를 해결하는 방법은 다음과 같습니다.

많은 웹 사이트는 알려진 사용자 에이전트 목록을 통해 웹 요청을 필터링하고 알 수 없는 또는 봇 사용자 에이전트에 대한 액세스를 거부합니다. Jsoup은 요청을 하면서 자바 버전을 사용자 에이전트 문자열로 사용합니다. 이는 봇의 범주에 속하며, 이로 인해 403 오류가 발생할 수 있습니다 - 액세스가 금지되었습니다.

이 문제를 해결하기 위한 간단한 해결책은 다음과 같이 Jsoup을 사용하여 웹 사이트에 연결할 때 사용자 에이전트 문자열을 지정하는 것입니다.

이렇다고 한다!

에러 해결 방법은

jsoup을 이용한 connect 요청 시 userAgent 값을 넣어주는 것이었다.

String url = "https://www.google.com/search?q=%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91+%EC%98%81%EC%96%B4%EB%A1%9C&oq=%EC%8A%A4%ED%81%AC%EB%9E%98%ED%95%91+%EC%98%81%EC%96%B4%EB%A1%9C&gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCDMyMzBqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8";
Connection connection = Jsoup.connect(url);

//specify user agent
connection.userAgent("Mozilla/5.0");

Document document = connection.get();

해결! 🥸